Nov 03 13:03:47.073070 ip-10-0-132-174 systemd[1]: Starting Kubernetes Kubelet... Nov 03 13:03:47.433735 ip-10-0-132-174 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:03:47.433735 ip-10-0-132-174 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 13:03:47.433735 ip-10-0-132-174 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:03:47.433735 ip-10-0-132-174 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:03:47.439345 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439258 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:47.439345 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439344 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:47.439345 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439349 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439352 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439355 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439358 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439361 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439370 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439373 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439376 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439378 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439381 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439384 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439386 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439389 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439392 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439395 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439397 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439400 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439403 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439405 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439408 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:47.439442 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439410 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439413 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439415 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439418 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439420 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439423 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439426 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439429 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439433 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439437 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439440 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439444 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439447 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439450 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439453 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439456 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439459 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439462 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439466 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:47.439933 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439468 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439471 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439474 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439477 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439479 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439482 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439485 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439488 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439490 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439493 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439496 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439499 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439501 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439504 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439507 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439509 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439512 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439515 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439517 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439520 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:47.440378 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439523 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439526 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439528 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439531 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439534 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439537 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439540 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439543 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439545 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439548 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439551 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439553 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439556 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439559 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439562 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439564 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439567 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439569 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439572 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:47.441176 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439574 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439577 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439580 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439582 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439586 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439590 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439593 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439596 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439599 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439601 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439604 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439606 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439610 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439612 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439615 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439618 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439621 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439624 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439627 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:47.441903 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439629 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439632 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439635 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439637 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439640 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439642 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.439645 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440681 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440707 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440712 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440719 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440724 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440729 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440733 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440737 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440741 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440745 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440750 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440754 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440758 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:47.442366 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440763 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440785 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440790 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440794 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440799 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440804 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440808 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440812 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440816 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440820 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440824 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440828 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440832 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440837 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440846 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440850 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440854 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440858 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440862 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440867 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:47.442916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440873 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440878 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440883 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440887 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440892 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440896 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440905 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440910 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440914 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440918 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440923 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440927 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440933 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440937 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440943 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440948 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440952 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440956 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440965 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:47.443408 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440969 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440974 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440979 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440983 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440988 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440992 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.440997 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441002 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441006 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441011 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441015 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441024 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441029 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441035 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441045 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441050 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441054 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441059 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441063 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:47.443876 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441067 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441072 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441076 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441080 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441085 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441093 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441098 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441102 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441106 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441110 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441114 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441118 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441122 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441126 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441130 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441134 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441139 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441143 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441151 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441155 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:47.444323 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441159 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441164 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441168 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441172 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441176 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441182 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441189 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441197 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441202 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441207 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441212 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441221 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441225 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441235 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.441240 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442117 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442126 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442129 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442133 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442135 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:47.444832 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442138 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442141 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442144 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442147 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442150 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442152 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442156 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442159 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442161 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442164 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442166 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442169 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442172 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442174 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442177 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442180 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442182 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442185 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442188 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442191 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:47.445311 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442194 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442196 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442199 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442201 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442203 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442206 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442209 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442212 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442215 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442218 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442220 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442223 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442225 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442228 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442230 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442233 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442235 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442238 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442240 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:47.445823 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442243 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442245 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442248 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442250 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442253 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442255 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442258 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442260 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442263 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442266 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442268 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442271 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442274 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442276 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442281 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442285 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442289 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442292 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:47.446306 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442295 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442298 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442301 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442303 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442306 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442308 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442311 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442314 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442317 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442319 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442322 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442324 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442327 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442330 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442332 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442335 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442337 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442340 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442343 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442347 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:47.446752 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442351 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442353 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442356 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442359 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442361 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442364 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442367 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442370 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442372 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442375 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442378 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442381 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442384 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442387 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442390 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442393 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442395 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442398 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442400 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442403 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:47.447245 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442406 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442408 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442414 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.442416 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443330 2585 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443341 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443349 2585 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443355 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443360 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443363 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443367 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443372 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443375 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443378 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443382 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443385 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443388 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443391 2585 flags.go:64] FLAG: --cgroup-root="" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443394 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443397 2585 flags.go:64] FLAG: --client-ca-file="" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443400 2585 flags.go:64] FLAG: --cloud-provider="external" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443403 2585 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443407 2585 flags.go:64] FLAG: --cluster-domain="" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443410 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 13:03:47.447766 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443414 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443418 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443421 2585 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443426 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443429 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443432 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443435 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443438 2585 flags.go:64] FLAG: --contention-profiling="false" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443441 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443444 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443448 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443451 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443456 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443458 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443461 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443465 2585 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443468 2585 flags.go:64] FLAG: --enable-server="true" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443471 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443477 2585 flags.go:64] FLAG: --event-burst="100" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443480 2585 flags.go:64] FLAG: --event-qps="50" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443484 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443487 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443490 2585 flags.go:64] FLAG: --eviction-hard="" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443494 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 13:03:47.448354 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443497 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443500 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443503 2585 flags.go:64] FLAG: --eviction-soft="" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443506 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443509 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443512 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443518 2585 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443521 2585 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443524 2585 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443527 2585 flags.go:64] FLAG: --feature-gates="" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443531 2585 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443534 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443537 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443541 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443544 2585 flags.go:64] FLAG: --healthz-port="10248" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443547 2585 flags.go:64] FLAG: --help="false" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443550 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443554 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443557 2585 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443560 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443563 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443567 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443570 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 13:03:47.448948 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443573 2585 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443576 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443579 2585 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443582 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443585 2585 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443588 2585 flags.go:64] FLAG: --kube-reserved="" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443591 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443594 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443597 2585 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443600 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443603 2585 flags.go:64] FLAG: --lock-file="" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443606 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443609 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443612 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443618 2585 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443624 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443627 2585 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443630 2585 flags.go:64] FLAG: --logging-format="text" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443633 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443636 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443639 2585 flags.go:64] FLAG: --manifest-url="" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443642 2585 flags.go:64] FLAG: --manifest-url-header="" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443647 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443650 2585 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443654 2585 flags.go:64] FLAG: --max-pods="110" Nov 03 13:03:47.449491 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443657 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443661 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443664 2585 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443667 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443669 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443672 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443675 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443683 2585 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443686 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443690 2585 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443693 2585 flags.go:64] FLAG: --pod-cidr="" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443696 2585 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443699 2585 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443702 2585 flags.go:64] FLAG: --pods-per-core="0" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443705 2585 flags.go:64] FLAG: --port="10250" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443708 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443711 2585 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e0a2f74ef738df33" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443714 2585 flags.go:64] FLAG: --qos-reserved="" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443717 2585 flags.go:64] FLAG: --read-only-port="10255" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443720 2585 flags.go:64] FLAG: --register-node="true" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443723 2585 flags.go:64] FLAG: --register-with-taints="" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443727 2585 flags.go:64] FLAG: --registry-burst="10" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443731 2585 flags.go:64] FLAG: --registry-qps="5" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443734 2585 flags.go:64] FLAG: --reserved-cpus="" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443737 2585 flags.go:64] FLAG: --reserved-memory="" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443741 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 13:03:47.450112 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443744 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443748 2585 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443751 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443754 2585 flags.go:64] FLAG: --runonce="false" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443757 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443760 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443763 2585 flags.go:64] FLAG: --seccomp-default="false" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443766 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443783 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443786 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443789 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443792 2585 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443796 2585 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443798 2585 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443802 2585 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443805 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443809 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443812 2585 flags.go:64] FLAG: --system-cgroups="" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443815 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443820 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443823 2585 flags.go:64] FLAG: --tls-cert-file="" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443826 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443830 2585 flags.go:64] FLAG: --tls-min-version="" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443833 2585 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443836 2585 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 13:03:47.450752 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443838 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443842 2585 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443845 2585 flags.go:64] FLAG: --v="2" Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443851 2585 flags.go:64] FLAG: --version="false" Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443855 2585 flags.go:64] FLAG: --vmodule="" Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443859 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.443863 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.443999 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444002 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444005 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444008 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444010 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444013 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444016 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444018 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444021 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444024 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444026 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444029 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444031 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444034 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:47.451373 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444037 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444039 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444042 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444045 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444048 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444051 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444054 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444057 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444060 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444062 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444065 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444067 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444070 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444072 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444077 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444080 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444082 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444085 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444088 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444091 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:47.451972 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444094 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444096 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444099 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444102 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444104 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444107 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444110 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444112 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444115 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444117 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444120 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444122 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444125 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444127 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444130 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444132 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444135 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444138 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444141 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444143 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:47.452452 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444146 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444148 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444151 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444153 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444156 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444158 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444163 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444166 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444168 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444171 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444175 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444177 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444180 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444182 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444185 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444188 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444190 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444193 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444195 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:47.452949 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444198 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444201 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444204 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444206 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444209 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444211 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444214 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444216 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444220 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444224 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444228 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444230 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444233 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444236 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444238 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444241 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444243 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444247 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444251 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:47.453410 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444255 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444258 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444261 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444264 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444268 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444271 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444274 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444276 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444279 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444281 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444284 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444287 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444290 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444293 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.444319 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444356 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:47.453878 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444361 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444364 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444367 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444369 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444372 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444375 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444377 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444380 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444383 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444386 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444389 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444391 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444394 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444396 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444399 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444401 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444404 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444408 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444410 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444413 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:47.454259 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444417 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444419 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444422 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444424 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444427 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444430 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444432 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444435 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444437 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444440 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444442 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444445 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444448 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444451 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444453 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444456 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444458 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444461 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444464 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444466 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:47.454739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444469 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444473 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444476 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444479 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444482 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444484 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444487 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444489 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444492 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444496 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444499 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444502 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444508 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444511 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444514 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444517 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444520 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444522 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444525 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:47.455249 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444527 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444532 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444535 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444539 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444542 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444544 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444547 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444550 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444552 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444555 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444557 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444560 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444562 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444582 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444586 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444591 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444594 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444597 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444599 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:47.455712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444602 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444604 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444607 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444611 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444613 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444616 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444620 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444622 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444625 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444628 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444630 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444633 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444636 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444638 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444641 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444644 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444646 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444649 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444652 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444654 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:47.456185 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444658 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444660 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444663 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444666 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444668 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444671 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.444673 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.451116 2585 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.451130 2585 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.451582 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.451588 2585 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451646 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451651 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451655 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451658 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451661 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451664 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451666 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451669 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:47.456697 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451672 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451674 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451677 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451679 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451682 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451685 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451688 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451690 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451693 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451695 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451698 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451701 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451703 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451705 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451708 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451711 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451714 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451716 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451719 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451721 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:47.457167 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451724 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451729 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451732 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451735 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451737 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451740 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451743 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451747 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451751 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451754 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451757 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451760 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451763 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451766 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451781 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451784 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451787 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451790 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451792 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:47.457624 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451795 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451798 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451800 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451803 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451805 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451808 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451811 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451813 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451816 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451818 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451821 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451824 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451826 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451829 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451832 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451836 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451839 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451842 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451845 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:47.458111 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451848 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451851 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451853 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451856 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451859 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451861 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451864 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451866 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451869 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451871 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451874 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451876 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451879 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451882 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451884 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451887 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451889 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451892 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451894 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451898 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:47.458586 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451903 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451906 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451909 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451911 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451914 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451917 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451920 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451923 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451926 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451929 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451932 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451934 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451937 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451939 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451942 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451944 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451947 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451949 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451952 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:47.459069 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.451955 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.451982 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452121 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452127 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452130 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452133 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452136 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452139 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452143 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452146 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452148 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452151 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452154 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452156 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452159 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452161 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:47.459539 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452164 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452166 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452169 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452172 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452175 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452178 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452180 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452184 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452186 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452189 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452192 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452194 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452197 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452199 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452203 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452207 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452210 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452213 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452216 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:47.459979 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452219 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452221 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452224 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452226 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452229 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452232 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452234 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452237 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452240 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452242 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452245 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452247 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452250 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452252 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452255 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452258 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452262 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452266 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452269 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:47.460447 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452271 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452274 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452277 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452279 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452282 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452285 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452287 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452290 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452292 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452295 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452297 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452300 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452302 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452305 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452307 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452310 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452313 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452315 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452318 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452320 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452323 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:47.460915 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452325 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452328 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452330 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452333 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452335 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452338 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452340 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452343 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452346 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452348 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452351 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452354 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452356 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452359 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452362 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452365 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452367 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452370 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452372 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452375 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:47.461419 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452377 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452380 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452382 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452385 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452387 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452390 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452392 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452395 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452397 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452399 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452402 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452404 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452407 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.452430 2585 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452462 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:03:47.461916 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452466 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452469 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452472 2585 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452474 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452477 2585 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452480 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452482 2585 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452485 2585 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452488 2585 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452492 2585 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452495 2585 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452498 2585 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452502 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452504 2585 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452507 2585 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452509 2585 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452512 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452515 2585 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452517 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:03:47.462266 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452520 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452523 2585 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452525 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452528 2585 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452531 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452533 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452536 2585 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452538 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452541 2585 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452544 2585 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452546 2585 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452548 2585 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452551 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452554 2585 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452557 2585 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452559 2585 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452562 2585 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452564 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452566 2585 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452569 2585 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:03:47.462712 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452571 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452574 2585 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452577 2585 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452580 2585 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452582 2585 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452587 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452590 2585 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452593 2585 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452595 2585 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452598 2585 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452600 2585 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452603 2585 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452605 2585 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452608 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452611 2585 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452613 2585 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452616 2585 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452619 2585 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452621 2585 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452624 2585 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:03:47.463340 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452626 2585 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452629 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452631 2585 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452634 2585 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452636 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452639 2585 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452642 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452644 2585 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452647 2585 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452649 2585 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452652 2585 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452655 2585 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452657 2585 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452660 2585 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452663 2585 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452665 2585 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452668 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452672 2585 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452675 2585 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:03:47.464206 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452679 2585 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452681 2585 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452684 2585 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452687 2585 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452689 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452692 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452695 2585 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452697 2585 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452700 2585 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452702 2585 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452705 2585 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452707 2585 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452710 2585 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452712 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452715 2585 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452717 2585 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452721 2585 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452726 2585 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452729 2585 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:03:47.464719 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452731 2585 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452734 2585 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452736 2585 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452739 2585 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452742 2585 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452744 2585 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452747 2585 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:47.452749 2585 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.452818 2585 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.454578 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.455364 2585 server.go:1014] "Starting client certificate rotation" Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.455463 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.456010 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.457206 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:03:47.465238 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.458565 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:03:47.470414 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.470400 2585 log.go:25] "Validated CRI v1 runtime API" Nov 03 13:03:47.473266 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.473251 2585 log.go:25] "Validated CRI v1 image API" Nov 03 13:03:47.474104 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.474081 2585 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 13:03:47.476993 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.476968 2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a9751310-f22a-441f-9fed-e1ddc205e8fa:/dev/nvme0n1p4 f484694d-4008-4563-84bc-8dd7427d2bfc:/dev/nvme0n1p3] Nov 03 13:03:47.477087 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.476992 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 13:03:47.482358 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.482237 2585 manager.go:217] Machine: {Timestamp:2026-11-03 13:03:47.480762021 +0000 UTC m=+0.297853077 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102307 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec210648c3ea5f491cff8be8b1aee627 SystemUUID:ec210648-c3ea-5f49-1cff-8be8b1aee627 BootID:8c76694b-7b45-4a7f-b919-8c9f990e729b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c9:dd:1d:c7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c9:dd:1d:c7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:8d:6a:31:29:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 13:03:47.482788 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.482762 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 13:03:47.482941 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.482918 2585 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 13:03:47.485669 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.485647 2585 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 13:03:47.485821 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.485670 2585 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-174.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 13:03:47.485867 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.485825 2585 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 13:03:47.485867 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.485834 2585 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 13:03:47.485916 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.485887 2585 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:03:47.485916 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.485905 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:03:47.486688 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.486679 2585 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 13:03:47.488959 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.488947 2585 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 13:03:47.489075 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.489065 2585 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 13:03:47.489691 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.489682 2585 kubelet.go:493] "Attempting to sync node with API server" Nov 03 13:03:47.489719 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.489696 2585 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 13:03:47.489719 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.489706 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 13:03:47.489719 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.489716 2585 kubelet.go:405] "Adding apiserver pod source" Nov 03 13:03:47.489841 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.489725 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 13:03:47.490089 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.490073 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:03:47.492177 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.492157 2585 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 13:03:47.496250 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.496237 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:03:47.496912 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.496900 2585 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 13:03:47.496945 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.496930 2585 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 13:03:47.497084 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497075 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 13:03:47.497123 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497090 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 13:03:47.497123 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497096 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 13:03:47.497123 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497102 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 13:03:47.497123 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497108 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 13:03:47.497123 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497113 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 13:03:47.497693 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497685 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 13:03:47.497721 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497694 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 13:03:47.497721 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497701 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 13:03:47.497721 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497708 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 13:03:47.497721 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497716 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 13:03:47.497857 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.497724 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 13:03:47.498639 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.498630 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 13:03:47.498673 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.498640 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 13:03:47.502449 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.502438 2585 server.go:1263] "Started kubelet" Nov 03 13:03:47.502524 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.502498 2585 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 13:03:47.503081 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.502530 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 13:03:47.503151 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.503119 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 13:03:47.503413 ip-10-0-132-174 systemd[1]: Started Kubernetes Kubelet. Nov 03 13:03:47.504751 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.504731 2585 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 13:03:47.506172 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.505159 2585 server.go:317] "Adding debug handlers to kubelet server" Nov 03 13:03:47.508055 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.507947 2585 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 13:03:47.511371 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.511353 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 13:03:47.512018 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.512006 2585 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 13:03:47.512107 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.512090 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:47.512626 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.512609 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 13:03:47.512626 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.512611 2585 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 13:03:47.512626 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.512637 2585 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 13:03:47.512845 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.512735 2585 reconstruct.go:97] "Volume reconstruction finished" Nov 03 13:03:47.512845 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.512744 2585 reconciler.go:29] "Reconciler: start to sync state" Nov 03 13:03:47.514403 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.514388 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 13:03:47.514403 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.514403 2585 factory.go:55] Registering systemd factory Nov 03 13:03:47.514529 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.514409 2585 factory.go:223] Registration of the systemd container factory successfully Nov 03 13:03:47.514639 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.514624 2585 factory.go:153] Registering CRI-O factory Nov 03 13:03:47.514680 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.514643 2585 factory.go:223] Registration of the crio container factory successfully Nov 03 13:03:47.514680 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.514660 2585 factory.go:103] Registering Raw factory Nov 03 13:03:47.514680 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.514673 2585 manager.go:1199] Started watching for new ooms in manager Nov 03 13:03:47.515469 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.515455 2585 manager.go:319] Starting recovery of all containers Nov 03 13:03:47.525287 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.525258 2585 manager.go:324] Recovery completed Nov 03 13:03:47.530656 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.530620 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:47.532524 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.532506 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:47.532601 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.532541 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:47.532601 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.532556 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:47.533123 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.533108 2585 cpu_manager.go:226] "Starting" policy="none" Nov 03 13:03:47.533123 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.533122 2585 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 13:03:47.533251 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.533129 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:47.533251 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.533143 2585 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 13:03:47.533807 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.533791 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:47.533882 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.533824 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:47.533882 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.533839 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:47.534378 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.534365 2585 policy_none.go:50] "Start" Nov 03 13:03:47.534439 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.534384 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 13:03:47.534439 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.534397 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 13:03:47.535402 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.535390 2585 policy_none.go:44] "Start" Nov 03 13:03:47.563412 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.563371 2585 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 13:03:47.566096 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.566080 2585 manager.go:341] "Starting Device Plugin manager" Nov 03 13:03:47.566178 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.566115 2585 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 13:03:47.566178 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.566127 2585 server.go:87] "Starting device plugin registration server" Nov 03 13:03:47.566358 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.566345 2585 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 13:03:47.566387 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.566359 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 13:03:47.566482 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.566463 2585 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 13:03:47.566575 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.566562 2585 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 13:03:47.566632 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.566577 2585 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 13:03:47.567142 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.567121 2585 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 13:03:47.567210 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.567174 2585 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:47.637413 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.637372 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-174.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 13:03:47.637413 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.637377 2585 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-174.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 13:03:47.638528 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.637377 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-174.us-west-2.compute.internal.18e48aad0ffb6bdb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 13:03:47.502410715 +0000 UTC m=+0.319501773,LastTimestamp:2026-11-03 13:03:47.502410715 +0000 UTC m=+0.319501773,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-174.us-west-2.compute.internal,}" Nov 03 13:03:47.642980 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.642954 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 13:03:47.644189 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.644163 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 13:03:47.644270 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.644194 2585 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 13:03:47.644270 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.644216 2585 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 13:03:47.644270 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.644253 2585 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 13:03:47.667327 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.667309 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:47.668196 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.668178 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:47.668264 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.668215 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:47.668264 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.668232 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:47.668264 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.668258 2585 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.701222 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.701146 2585 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-174.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 13:03:47.701332 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.701163 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-174.us-west-2.compute.internal.18e48aad11c6e590 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-174.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 13:03:47.532522896 +0000 UTC m=+0.349613953,LastTimestamp:2026-11-03 13:03:47.532522896 +0000 UTC m=+0.349613953,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-174.us-west-2.compute.internal,}" Nov 03 13:03:47.701332 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.701288 2585 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 13:03:47.744864 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.744835 2585 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal"] Nov 03 13:03:47.744984 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.744899 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:47.745817 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.745802 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:47.745878 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.745833 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:47.745878 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.745843 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:47.747331 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.747318 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:47.747479 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.747464 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.747516 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.747500 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:47.748878 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.748860 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:47.748978 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.748898 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:47.748978 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.748912 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:47.748978 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.748865 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:47.749110 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.748983 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:47.749110 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.748995 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:47.750158 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.750141 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.750250 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.750171 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:47.751081 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.751063 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:47.751194 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.751105 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:47.751194 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.751119 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:47.773556 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.773534 2585 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.777007 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.776987 2585 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.789999 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.789979 2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 13:03:47.790090 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.790035 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zkl2h" Nov 03 13:03:47.794156 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.794127 2585 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 13:03:47.794259 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.794192 2585 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 03 13:03:47.794259 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.794171 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-174.us-west-2.compute.internal.18e48aad11c748f3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-174.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 13:03:47.532548339 +0000 UTC m=+0.349639397,LastTimestamp:2026-11-03 13:03:47.532548339 +0000 UTC m=+0.349639397,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-174.us-west-2.compute.internal,}" Nov 03 13:03:47.849092 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.849062 2585 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.913924 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.913897 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.914109 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.913942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.914109 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:47.913958 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:47.935948 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:47.935928 2585 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.014916 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.014847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.014916 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.014875 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.014916 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.014892 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.015101 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.014962 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.015101 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.014972 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.015101 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.014962 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.050018 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.049977 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:48.051116 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.051096 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:48.051224 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.051133 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:48.051224 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.051147 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:48.051224 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.051182 2585 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.075903 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.075875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.080428 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.080414 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.104418 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.104395 2585 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:48.104418 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.104421 2585 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-174.us-west-2.compute.internal\": node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:48.209942 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.209917 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:48.310658 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.310577 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:48.411154 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.410955 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:48.455661 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.455632 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 13:03:48.456127 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.455827 2585 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a16aeb6c627d247dd9f516a896748b1c-7174c5b4c9fb5a11.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-132-174.us-west-2.compute.internal.18e48aad11c748f3\": read tcp 10.0.132.174:60520->3.138.57.107:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-132-174.us-west-2.compute.internal.18e48aad11c748f3 default 4987 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-174.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-174.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-174.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 13:03:47 +0000 UTC,LastTimestamp:2026-11-03 13:03:47.745837841 +0000 UTC m=+0.562928896,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-174.us-west-2.compute.internal,}" Nov 03 13:03:48.511399 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.511364 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:48.613478 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.613384 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:48.713988 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.713948 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:48.793703 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.793622 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 12:58:47 +0000 UTC" deadline="2028-08-11 01:07:00.251830732 +0000 UTC" Nov 03 13:03:48.793703 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.793695 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15516h3m11.458141259s" Nov 03 13:03:48.814994 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.814969 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:48.835176 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.835148 2585 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:03:48.836219 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.836202 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:03:48.836291 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.836237 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:03:48.836291 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.836250 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:03:48.915147 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:48.915123 2585 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:03:48.937730 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:48.937705 2585 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:49.013120 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.013085 2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:49.017609 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.017591 2585 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:49.076895 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.076863 2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:03:49.077758 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.077744 2585 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:03:49.136267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.136246 2585 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:03:49.163542 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.163517 2585 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:49.230768 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.230701 2585 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:03:49.250394 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.250359 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac4092c55873e071f156a04a17c98a5.slice/crio-72778de9f271e583a7afed12182177aa39d73fb9aed049cb862ba0d20210617e WatchSource:0}: Error finding container 72778de9f271e583a7afed12182177aa39d73fb9aed049cb862ba0d20210617e: Status 404 returned error can't find the container with id 72778de9f271e583a7afed12182177aa39d73fb9aed049cb862ba0d20210617e Nov 03 13:03:49.250828 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.250802 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747e2de570be6f488466605f00369eb2.slice/crio-1fb1aebcff1ab1345a58c682d1bdd3ea96b0ddb540f8a968e6ecdc67e9e21fa7 WatchSource:0}: Error finding container 1fb1aebcff1ab1345a58c682d1bdd3ea96b0ddb540f8a968e6ecdc67e9e21fa7: Status 404 returned error can't find the container with id 1fb1aebcff1ab1345a58c682d1bdd3ea96b0ddb540f8a968e6ecdc67e9e21fa7 Nov 03 13:03:49.257587 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.257569 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:03:49.491417 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.491327 2585 apiserver.go:52] "Watching apiserver" Nov 03 13:03:49.513534 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.513505 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 13:03:49.589035 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.589004 2585 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 13:03:49.589310 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.589291 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-qm2ff","openshift-ovn-kubernetes/ovnkube-node-5sjz9","kube-system/konnectivity-agent-pjnlk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b","openshift-cluster-node-tuning-operator/tuned-mjx8m","openshift-multus/multus-j99cl","openshift-network-diagnostics/network-check-target-hfbh8","openshift-network-operator/iptables-alerter-k8dl6","openshift-multus/network-metrics-daemon-r8f65","kube-system/global-pull-secret-syncer-ncrq2","openshift-image-registry/node-ca-tgrzf","kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal","openshift-dns/node-resolver-5zlvj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal"] Nov 03 13:03:49.591406 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.591389 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:49.591475 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.591458 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:03:49.592392 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.592379 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.593292 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.593278 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:03:49.594337 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.594322 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.595320 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.595304 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.596154 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.596142 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.597357 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.597339 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:49.597464 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.597390 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:03:49.598476 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.598458 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.599541 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.599520 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.600740 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.600721 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:49.600842 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.600821 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:03:49.601784 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.601755 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.602652 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.602633 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.613322 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.613306 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 13:03:49.624223 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624202 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-sys-fs\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.624223 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624228 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.624382 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624248 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-kube-api-access-ks6h6\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.624382 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624264 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-dbus\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:49.624382 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624321 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnd4j\" (UniqueName: \"kubernetes.io/projected/d86486e7-2eb7-4200-a35b-8025870fc018-kube-api-access-nnd4j\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.624496 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624381 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.624496 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624423 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-config\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.624496 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624441 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-var-lib-kubelet\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.624578 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624518 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cnibin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.624578 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-conf-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.624578 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624568 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-registration-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.624663 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624582 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:49.624663 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624600 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.624663 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624634 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnrpn\" (UniqueName: \"kubernetes.io/projected/8b2a6fcd-a217-4a03-994a-9261ed191c64-kube-api-access-nnrpn\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.624837 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624687 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-socket-dir-parent\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.624837 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624727 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-kubelet\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.624837 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624757 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-daemon-config\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.624837 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624799 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-system-cni-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.624837 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-sys\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.625048 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624858 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-kubelet\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.625048 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624883 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-var-lib-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.625048 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624923 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-log-socket\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.625048 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624974 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-kubelet-config\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:49.625048 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.624990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-netd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.625220 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625073 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-etc-selinux\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.625220 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625107 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-ovn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.625220 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625132 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.625220 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625183 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-tmp\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.625393 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625217 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-slash\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.625393 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625242 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysconfig\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.625393 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625277 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-conf\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.625393 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625301 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-run\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.625393 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625362 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-bin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.625393 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625390 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-script-lib\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.625652 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625422 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb630b3-a283-4fa8-9887-1202daceae1a-host\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.625652 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vwsq\" (UniqueName: \"kubernetes.io/projected/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-kube-api-access-7vwsq\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.625652 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48d92b6c-4577-458c-b6a7-7b11300ca6b1-hosts-file\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.625801 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625787 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.625843 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625819 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm6g2\" (UniqueName: \"kubernetes.io/projected/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-kube-api-access-zm6g2\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.625843 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625836 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:49.625917 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625850 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-agent-certs\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:03:49.625917 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625864 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-kubernetes\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.625917 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625883 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-lib-modules\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.626038 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625941 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-system-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.626038 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.625988 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.626038 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626025 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:49.626140 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626054 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-netns\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.626140 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626083 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.626140 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626099 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-env-overrides\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.626140 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626117 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtzxj\" (UniqueName: \"kubernetes.io/projected/cf9f8413-6613-4db2-9dd1-20742c2af7ed-kube-api-access-qtzxj\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:49.626267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cnibin\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.626267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626160 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-systemd-units\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.626267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-node-log\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.626267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626190 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d92b6c-4577-458c-b6a7-7b11300ca6b1-tmp-dir\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.626267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626237 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-netns\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.626399 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-multus-certs\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.626399 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626312 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.626399 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626334 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-socket-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.626399 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prpcz\" (UniqueName: \"kubernetes.io/projected/1bb630b3-a283-4fa8-9887-1202daceae1a-kube-api-access-prpcz\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.626399 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626377 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.626399 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626398 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bb630b3-a283-4fa8-9887-1202daceae1a-serviceca\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.626562 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626436 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwzw9\" (UniqueName: \"kubernetes.io/projected/48d92b6c-4577-458c-b6a7-7b11300ca6b1-kube-api-access-hwzw9\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.626562 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626456 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-modprobe-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.626562 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626470 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-systemd\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.626562 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626483 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-k8s-cni-cncf-io\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.626562 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626499 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-multus\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.626562 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626551 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-etc-kubernetes\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.626798 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626567 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-bin\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.626798 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626581 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.626798 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626618 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cni-binary-copy\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.626798 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626642 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-hostroot\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.626798 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626662 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-systemd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.626798 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626690 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-etc-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.626798 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626726 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvswn\" (UniqueName: \"kubernetes.io/projected/f9f28189-a808-48a4-b2da-d1950b84c7d0-kube-api-access-mvswn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.627107 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626833 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-iptables-alerter-script\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.627107 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626863 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-host-slash\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.627107 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626906 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-host\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.627107 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.626971 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-tuned\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.627107 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.627008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.627107 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.627035 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-konnectivity-ca\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:03:49.627107 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.627077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-device-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.642257 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.642224 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 13:03:49.642257 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.642235 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 13:03:49.642257 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.642251 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 13:03:49.642696 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.642678 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-sctm8\"" Nov 03 13:03:49.642813 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.642738 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 13:03:49.642813 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.642746 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8v28q\"" Nov 03 13:03:49.643149 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.643133 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 13:03:49.648097 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.648057 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerStarted","Data":"1fb1aebcff1ab1345a58c682d1bdd3ea96b0ddb540f8a968e6ecdc67e9e21fa7"} Nov 03 13:03:49.649116 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.649099 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"3ac4092c55873e071f156a04a17c98a5","Type":"ContainerStarted","Data":"72778de9f271e583a7afed12182177aa39d73fb9aed049cb862ba0d20210617e"} Nov 03 13:03:49.680379 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680339 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5ntlc\"" Nov 03 13:03:49.680379 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680326 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 13:03:49.680379 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680338 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 13:03:49.680638 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680457 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-85kzz\"" Nov 03 13:03:49.680638 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680503 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 13:03:49.680638 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680554 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 13:03:49.680638 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680556 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:03:49.680638 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680563 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 13:03:49.680638 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680604 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 13:03:49.680856 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680660 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s6pph\"" Nov 03 13:03:49.680856 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680708 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 13:03:49.680856 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680731 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 13:03:49.680856 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.680767 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 13:03:49.683949 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.683935 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lbs9x\"" Nov 03 13:03:49.684277 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.684261 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 13:03:49.684326 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.684274 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5wxnl\"" Nov 03 13:03:49.684326 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.684304 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g4lr6\"" Nov 03 13:03:49.686463 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.686451 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 13:03:49.687870 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.687855 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l9fl5\"" Nov 03 13:03:49.687927 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.687871 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 13:03:49.687927 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.687900 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 13:03:49.689781 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.689752 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 13:03:49.689843 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.689758 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 13:03:49.689843 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.689798 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 13:03:49.689843 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.689800 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:03:49.689843 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.689809 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 13:03:49.689843 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.689840 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 13:03:49.690015 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.689925 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 13:03:49.690015 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.689947 2585 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 13:03:49.690015 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.689988 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m66mz" Nov 03 13:03:49.690789 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.690764 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 13:03:49.727854 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.727989 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727856 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:49.727989 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727874 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-netns\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.727989 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727908 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.727989 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727925 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-env-overrides\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.727989 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727940 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzxj\" (UniqueName: \"kubernetes.io/projected/cf9f8413-6613-4db2-9dd1-20742c2af7ed-kube-api-access-qtzxj\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:49.727989 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727939 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.727989 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727952 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-netns\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.727989 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727963 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.727976 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cnibin\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.728010 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728042 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cnibin\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.728084 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret podName:89e41adf-23e2-4eaa-a335-59002579bdce nodeName:}" failed. No retries permitted until 2026-11-03 13:03:50.228061236 +0000 UTC m=+3.045152298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret") pod "global-pull-secret-syncer-ncrq2" (UID: "89e41adf-23e2-4eaa-a335-59002579bdce") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728111 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-systemd-units\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728141 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-node-log\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d92b6c-4577-458c-b6a7-7b11300ca6b1-tmp-dir\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728172 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-netns\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728187 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-multus-certs\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728208 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728228 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-socket-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728137 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-systemd-units\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728251 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpcz\" (UniqueName: \"kubernetes.io/projected/1bb630b3-a283-4fa8-9887-1202daceae1a-kube-api-access-prpcz\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728263 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-netns\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728271 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-multus-certs\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728277 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.728298 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728302 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bb630b3-a283-4fa8-9887-1202daceae1a-serviceca\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728304 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-node-log\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728328 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzw9\" (UniqueName: \"kubernetes.io/projected/48d92b6c-4577-458c-b6a7-7b11300ca6b1-kube-api-access-hwzw9\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-modprobe-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728361 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-socket-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728379 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-systemd\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728403 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-k8s-cni-cncf-io\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728427 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-multus\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728437 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728448 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-etc-kubernetes\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728463 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-env-overrides\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728469 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-bin\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728522 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728536 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-bin\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728556 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cni-binary-copy\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728583 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-hostroot\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728583 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-k8s-cni-cncf-io\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728588 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d92b6c-4577-458c-b6a7-7b11300ca6b1-tmp-dir\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.729073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728605 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-systemd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728633 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-etc-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728637 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-etc-kubernetes\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728648 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-systemd\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728658 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvswn\" (UniqueName: \"kubernetes.io/projected/f9f28189-a808-48a4-b2da-d1950b84c7d0-kube-api-access-mvswn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728658 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-multus\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728589 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-modprobe-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728687 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-iptables-alerter-script\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728712 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-host-slash\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728715 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-hostroot\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728733 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-host\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728755 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-tuned\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728761 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-systemd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728799 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728818 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-etc-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728826 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-konnectivity-ca\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728849 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-device-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728857 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-host\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.729848 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-sys-fs\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728896 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728735 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728896 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728934 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-kube-api-access-ks6h6\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728947 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-host-slash\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728959 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-dbus\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728969 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd4j\" (UniqueName: \"kubernetes.io/projected/d86486e7-2eb7-4200-a35b-8025870fc018-kube-api-access-nnd4j\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728988 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-sys-fs\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.728995 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729014 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729021 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-config\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729044 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-var-lib-kubelet\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729058 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cnibin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729073 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-conf-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729088 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-registration-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.730560 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729102 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729120 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729138 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-dbus\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729137 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrpn\" (UniqueName: \"kubernetes.io/projected/8b2a6fcd-a217-4a03-994a-9261ed191c64-kube-api-access-nnrpn\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729178 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-socket-dir-parent\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729193 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-kubelet\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729197 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729205 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cnibin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729209 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-daemon-config\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729256 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-kubelet\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729260 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-device-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729258 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-var-lib-kubelet\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729302 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-system-cni-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729323 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-registration-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729327 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-sys\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729334 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-socket-dir-parent\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729344 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-conf-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729362 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-kubelet\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.731267 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729373 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-sys\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.729385 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729399 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-system-cni-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-var-lib-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729425 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-kubelet\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.729447 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:03:50.229431708 +0000 UTC m=+3.046522770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729466 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-log-socket\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729470 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bb630b3-a283-4fa8-9887-1202daceae1a-serviceca\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729472 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-var-lib-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729492 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-log-socket\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729496 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-kubelet-config\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729528 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-kubelet-config\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-netd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-etc-selinux\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729605 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-netd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729631 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-ovn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729657 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-ovn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732103 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729734 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-tmp\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729794 2585 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729820 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-slash\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729869 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cni-binary-copy\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729969 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-etc-selinux\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729975 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-config\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.729801 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-slash\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730201 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysconfig\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730254 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-conf\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730270 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysconfig\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730308 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-run\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730330 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-bin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730355 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-script-lib\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730361 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-run\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb630b3-a283-4fa8-9887-1202daceae1a-host\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vwsq\" (UniqueName: \"kubernetes.io/projected/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-kube-api-access-7vwsq\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730431 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48d92b6c-4577-458c-b6a7-7b11300ca6b1-hosts-file\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.732932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730454 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730519 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48d92b6c-4577-458c-b6a7-7b11300ca6b1-hosts-file\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730518 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-conf\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730551 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-iptables-alerter-script\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730603 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-bin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730624 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb630b3-a283-4fa8-9887-1202daceae1a-host\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730635 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm6g2\" (UniqueName: \"kubernetes.io/projected/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-kube-api-access-zm6g2\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730675 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730706 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-agent-certs\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730732 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-kubernetes\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730755 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-lib-modules\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730819 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-script-lib\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-system-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730867 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-lib-modules\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730869 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-kubernetes\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.730930 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-system-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.731107 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-konnectivity-ca\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:03:49.733573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.731157 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-daemon-config\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.734044 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.731242 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.734044 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.731363 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.734044 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.733456 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.734044 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.733458 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-tuned\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.734044 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.733488 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-tmp\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.734044 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.733657 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-agent-certs\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:03:49.782719 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.782494 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpcz\" (UniqueName: \"kubernetes.io/projected/1bb630b3-a283-4fa8-9887-1202daceae1a-kube-api-access-prpcz\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.782719 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.782633 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzxj\" (UniqueName: \"kubernetes.io/projected/cf9f8413-6613-4db2-9dd1-20742c2af7ed-kube-api-access-qtzxj\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:49.782952 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.782743 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:49.782952 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.782780 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:49.782952 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.782808 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nmhgn for pod openshift-network-diagnostics/network-check-target-hfbh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:49.782952 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.782867 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd4j\" (UniqueName: \"kubernetes.io/projected/d86486e7-2eb7-4200-a35b-8025870fc018-kube-api-access-nnd4j\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.782952 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:49.782905 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn podName:a7dc614f-3d12-4003-b572-8dbcbfcf7b6b nodeName:}" failed. No retries permitted until 2026-11-03 13:03:50.282875946 +0000 UTC m=+3.099967009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nmhgn" (UniqueName: "kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn") pod "network-check-target-hfbh8" (UID: "a7dc614f-3d12-4003-b572-8dbcbfcf7b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:49.783213 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.783038 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwzw9\" (UniqueName: \"kubernetes.io/projected/48d92b6c-4577-458c-b6a7-7b11300ca6b1-kube-api-access-hwzw9\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.783213 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.783051 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-kube-api-access-ks6h6\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.783213 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.783079 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvswn\" (UniqueName: \"kubernetes.io/projected/f9f28189-a808-48a4-b2da-d1950b84c7d0-kube-api-access-mvswn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.783362 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.783319 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrpn\" (UniqueName: \"kubernetes.io/projected/8b2a6fcd-a217-4a03-994a-9261ed191c64-kube-api-access-nnrpn\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.784287 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.784266 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vwsq\" (UniqueName: \"kubernetes.io/projected/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-kube-api-access-7vwsq\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.784655 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.784635 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm6g2\" (UniqueName: \"kubernetes.io/projected/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-kube-api-access-zm6g2\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.902820 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.902783 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:03:49.908700 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.908588 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:03:49.910874 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.910850 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d92b6c_4577_458c_b6a7_7b11300ca6b1.slice/crio-3ba7aa05fc2b28f32027396cda92af8499002e51a3e9b85eb294808a56270fb8 WatchSource:0}: Error finding container 3ba7aa05fc2b28f32027396cda92af8499002e51a3e9b85eb294808a56270fb8: Status 404 returned error can't find the container with id 3ba7aa05fc2b28f32027396cda92af8499002e51a3e9b85eb294808a56270fb8 Nov 03 13:03:49.916150 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.916127 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91090c0c_bbfd_4e9a_b7f0_4bf7e50027e8.slice/crio-e6fd964a7cdff5d40e64edc030cace79a4941c44ee1931f68ac96bd8628141cb WatchSource:0}: Error finding container e6fd964a7cdff5d40e64edc030cace79a4941c44ee1931f68ac96bd8628141cb: Status 404 returned error can't find the container with id e6fd964a7cdff5d40e64edc030cace79a4941c44ee1931f68ac96bd8628141cb Nov 03 13:03:49.916692 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.916676 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:03:49.920764 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.920743 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:03:49.923616 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.923595 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86486e7_2eb7_4200_a35b_8025870fc018.slice/crio-d45df744d7ce685c3686ce232a54f70ef8a7a0f294edb4d561ad8baaedc13944 WatchSource:0}: Error finding container d45df744d7ce685c3686ce232a54f70ef8a7a0f294edb4d561ad8baaedc13944: Status 404 returned error can't find the container with id d45df744d7ce685c3686ce232a54f70ef8a7a0f294edb4d561ad8baaedc13944 Nov 03 13:03:49.925569 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.925547 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j99cl" Nov 03 13:03:49.927929 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.927911 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2a6fcd_a217_4a03_994a_9261ed191c64.slice/crio-3a6c044648aa4c53fd3ea6e903b73156e16739719612ccebb227ec363c881683 WatchSource:0}: Error finding container 3a6c044648aa4c53fd3ea6e903b73156e16739719612ccebb227ec363c881683: Status 404 returned error can't find the container with id 3a6c044648aa4c53fd3ea6e903b73156e16739719612ccebb227ec363c881683 Nov 03 13:03:49.933739 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.933719 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed56df7_b752_4f2d_b941_0b7df5aa6df9.slice/crio-caec9813de1596f33731e527d0e07328d7f9818c6240c19d6731d636a7b0f14b WatchSource:0}: Error finding container caec9813de1596f33731e527d0e07328d7f9818c6240c19d6731d636a7b0f14b: Status 404 returned error can't find the container with id caec9813de1596f33731e527d0e07328d7f9818c6240c19d6731d636a7b0f14b Nov 03 13:03:49.936590 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.936573 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:03:49.943425 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.943405 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4c7dc1_2eea_456e_bc9a_b6d02814657f.slice/crio-8108f21a9c6fda01f5de299b4d89ec4742abc7f12f2850d8985c560bdb7b7665 WatchSource:0}: Error finding container 8108f21a9c6fda01f5de299b4d89ec4742abc7f12f2850d8985c560bdb7b7665: Status 404 returned error can't find the container with id 8108f21a9c6fda01f5de299b4d89ec4742abc7f12f2850d8985c560bdb7b7665 Nov 03 13:03:49.946463 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.946444 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:03:49.947452 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.947437 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:03:49.950194 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:49.950179 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:03:49.954068 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.953998 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f28189_a808_48a4_b2da_d1950b84c7d0.slice/crio-5f4950b203312825d4741862e4ed5d84fbac74b94028dab9c710e68b8733ddad WatchSource:0}: Error finding container 5f4950b203312825d4741862e4ed5d84fbac74b94028dab9c710e68b8733ddad: Status 404 returned error can't find the container with id 5f4950b203312825d4741862e4ed5d84fbac74b94028dab9c710e68b8733ddad Nov 03 13:03:49.956808 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.956784 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb630b3_a283_4fa8_9887_1202daceae1a.slice/crio-fdad73a7eea44c8258a8c910b3198dbcc478f6aa231fc2ac9c9bb0b032b1f668 WatchSource:0}: Error finding container fdad73a7eea44c8258a8c910b3198dbcc478f6aa231fc2ac9c9bb0b032b1f668: Status 404 returned error can't find the container with id fdad73a7eea44c8258a8c910b3198dbcc478f6aa231fc2ac9c9bb0b032b1f668 Nov 03 13:03:49.959454 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:03:49.959436 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0e9fa9_e8f0_4e70_ae9b_3fae77f8c07c.slice/crio-268e0a0f4b5f525db7154cc7aaeb9342b395c96d17370c0d91cc03bccb8dd516 WatchSource:0}: Error finding container 268e0a0f4b5f525db7154cc7aaeb9342b395c96d17370c0d91cc03bccb8dd516: Status 404 returned error can't find the container with id 268e0a0f4b5f525db7154cc7aaeb9342b395c96d17370c0d91cc03bccb8dd516 Nov 03 13:03:50.235975 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.235942 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:50.236133 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.236001 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:50.236133 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:50.236094 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:50.236133 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:50.236096 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:50.236247 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:50.236142 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret podName:89e41adf-23e2-4eaa-a335-59002579bdce nodeName:}" failed. No retries permitted until 2026-11-03 13:03:51.236128963 +0000 UTC m=+4.053220020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret") pod "global-pull-secret-syncer-ncrq2" (UID: "89e41adf-23e2-4eaa-a335-59002579bdce") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:50.236247 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:50.236154 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:03:51.236148102 +0000 UTC m=+4.053239145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:50.336707 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.336674 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:50.336913 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:50.336863 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:50.336913 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:50.336889 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:50.336913 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:50.336902 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nmhgn for pod openshift-network-diagnostics/network-check-target-hfbh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:50.337062 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:50.336961 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn podName:a7dc614f-3d12-4003-b572-8dbcbfcf7b6b nodeName:}" failed. No retries permitted until 2026-11-03 13:03:51.336942361 +0000 UTC m=+4.154033421 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmhgn" (UniqueName: "kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn") pod "network-check-target-hfbh8" (UID: "a7dc614f-3d12-4003-b572-8dbcbfcf7b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:50.651931 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.651894 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tgrzf" event={"ID":"1bb630b3-a283-4fa8-9887-1202daceae1a","Type":"ContainerStarted","Data":"fdad73a7eea44c8258a8c910b3198dbcc478f6aa231fc2ac9c9bb0b032b1f668"} Nov 03 13:03:50.653301 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.653265 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerStarted","Data":"5f4950b203312825d4741862e4ed5d84fbac74b94028dab9c710e68b8733ddad"} Nov 03 13:03:50.654585 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.654556 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" event={"ID":"ba4c7dc1-2eea-456e-bc9a-b6d02814657f","Type":"ContainerStarted","Data":"8108f21a9c6fda01f5de299b4d89ec4742abc7f12f2850d8985c560bdb7b7665"} Nov 03 13:03:50.655844 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.655816 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" event={"ID":"8b2a6fcd-a217-4a03-994a-9261ed191c64","Type":"ContainerStarted","Data":"3a6c044648aa4c53fd3ea6e903b73156e16739719612ccebb227ec363c881683"} Nov 03 13:03:50.657155 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.657138 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" event={"ID":"d86486e7-2eb7-4200-a35b-8025870fc018","Type":"ContainerStarted","Data":"d45df744d7ce685c3686ce232a54f70ef8a7a0f294edb4d561ad8baaedc13944"} Nov 03 13:03:50.658470 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.658449 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5zlvj" event={"ID":"48d92b6c-4577-458c-b6a7-7b11300ca6b1","Type":"ContainerStarted","Data":"3ba7aa05fc2b28f32027396cda92af8499002e51a3e9b85eb294808a56270fb8"} Nov 03 13:03:50.659748 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.659726 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8dl6" event={"ID":"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c","Type":"ContainerStarted","Data":"268e0a0f4b5f525db7154cc7aaeb9342b395c96d17370c0d91cc03bccb8dd516"} Nov 03 13:03:50.661038 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.661016 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j99cl" event={"ID":"8ed56df7-b752-4f2d-b941-0b7df5aa6df9","Type":"ContainerStarted","Data":"caec9813de1596f33731e527d0e07328d7f9818c6240c19d6731d636a7b0f14b"} Nov 03 13:03:50.662258 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.662234 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pjnlk" event={"ID":"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8","Type":"ContainerStarted","Data":"e6fd964a7cdff5d40e64edc030cace79a4941c44ee1931f68ac96bd8628141cb"} Nov 03 13:03:50.692950 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.692904 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 12:58:49 +0000 UTC" deadline="2028-05-26 10:51:31.534407552 +0000 UTC" Nov 03 13:03:50.692950 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:50.692942 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13677h47m40.841469596s" Nov 03 13:03:51.243708 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:51.243672 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:51.243910 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:51.243751 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:51.243910 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.243893 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:51.244026 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.243932 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:51.244026 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.243970 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret podName:89e41adf-23e2-4eaa-a335-59002579bdce nodeName:}" failed. No retries permitted until 2026-11-03 13:03:53.243948606 +0000 UTC m=+6.061039667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret") pod "global-pull-secret-syncer-ncrq2" (UID: "89e41adf-23e2-4eaa-a335-59002579bdce") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:51.244026 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.243993 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:03:53.243982037 +0000 UTC m=+6.061073080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:51.344429 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:51.344391 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:51.344644 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.344623 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:51.344705 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.344652 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:51.344705 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.344665 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nmhgn for pod openshift-network-diagnostics/network-check-target-hfbh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:51.344855 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.344731 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn podName:a7dc614f-3d12-4003-b572-8dbcbfcf7b6b nodeName:}" failed. No retries permitted until 2026-11-03 13:03:53.344705856 +0000 UTC m=+6.161796923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmhgn" (UniqueName: "kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn") pod "network-check-target-hfbh8" (UID: "a7dc614f-3d12-4003-b572-8dbcbfcf7b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:51.647487 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:51.644919 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:51.647487 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:51.644919 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:51.647487 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.645060 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:03:51.647487 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:51.645115 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:51.647487 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.645209 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:03:51.647487 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:51.645301 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:03:51.693131 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:51.693074 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 12:58:49 +0000 UTC" deadline="2028-06-05 05:08:35.784850603 +0000 UTC" Nov 03 13:03:51.693131 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:51.693105 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13912h4m44.091748236s" Nov 03 13:03:53.265657 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.265617 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:53.266290 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.265701 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:53.266290 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.265880 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:53.266290 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.265932 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret podName:89e41adf-23e2-4eaa-a335-59002579bdce nodeName:}" failed. No retries permitted until 2026-11-03 13:03:57.265919059 +0000 UTC m=+10.083010117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret") pod "global-pull-secret-syncer-ncrq2" (UID: "89e41adf-23e2-4eaa-a335-59002579bdce") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:53.266290 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.265975 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:53.266290 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.266010 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:03:57.265998145 +0000 UTC m=+10.083089204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:53.367281 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.366812 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:53.367281 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.367061 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:53.367281 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.367083 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:53.367281 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.367097 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nmhgn for pod openshift-network-diagnostics/network-check-target-hfbh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:53.367281 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.367157 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn podName:a7dc614f-3d12-4003-b572-8dbcbfcf7b6b nodeName:}" failed. No retries permitted until 2026-11-03 13:03:57.367137251 +0000 UTC m=+10.184228315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmhgn" (UniqueName: "kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn") pod "network-check-target-hfbh8" (UID: "a7dc614f-3d12-4003-b572-8dbcbfcf7b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:53.645272 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.644884 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:53.645272 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.645016 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:03:53.645272 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.645088 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:53.645272 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.645162 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:03:53.645727 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.645604 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:53.645727 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:53.645696 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:03:53.685304 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.685265 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" event={"ID":"ba4c7dc1-2eea-456e-bc9a-b6d02814657f","Type":"ContainerStarted","Data":"7f80d8a8055a2769099f9fa76d8333177d3dde24a2f25c574cee54d88bfc5e09"} Nov 03 13:03:53.691541 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.691507 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"3ac4092c55873e071f156a04a17c98a5","Type":"ContainerStarted","Data":"5e0af216ce38053eef0872c0b06eb860018a48d37240953dc242023fd46d625c"} Nov 03 13:03:53.694400 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.694372 2585 generic.go:356] "Generic (PLEG): container finished" podID="747e2de570be6f488466605f00369eb2" containerID="f439e359a8800a2cf56f86288a9a5c4c3e95ff0c86530dfd5ecf56c318491213" exitCode=0 Nov 03 13:03:53.694539 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:53.694417 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerDied","Data":"f439e359a8800a2cf56f86288a9a5c4c3e95ff0c86530dfd5ecf56c318491213"} Nov 03 13:03:54.096456 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:54.095159 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" podStartSLOduration=5.095138671 podStartE2EDuration="5.095138671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:53.98913638 +0000 UTC m=+6.806227667" watchObservedRunningTime="2026-11-03 13:03:54.095138671 +0000 UTC m=+6.912229739" Nov 03 13:03:54.710956 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:54.710117 2585 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="7f80d8a8055a2769099f9fa76d8333177d3dde24a2f25c574cee54d88bfc5e09" exitCode=0 Nov 03 13:03:54.710956 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:54.710207 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" event={"ID":"ba4c7dc1-2eea-456e-bc9a-b6d02814657f","Type":"ContainerDied","Data":"7f80d8a8055a2769099f9fa76d8333177d3dde24a2f25c574cee54d88bfc5e09"} Nov 03 13:03:54.715281 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:54.715250 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerStarted","Data":"a309afd6a252b66a1129aa5ccf3931a9ec2f2cc20a4bf303ed8de3b352c062ff"} Nov 03 13:03:55.645005 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:55.644875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:55.645151 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:55.645010 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:03:55.645151 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:55.645086 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:55.645268 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:55.645157 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:03:55.645320 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:55.645286 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:55.647518 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:55.645373 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:03:55.721039 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:55.720994 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pjnlk" event={"ID":"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8","Type":"ContainerStarted","Data":"68ea36754db94deb71c94a18ced5bc8f18d3177ba2a08addf2f97c77b63c1c25"} Nov 03 13:03:55.725195 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:55.725168 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tgrzf" event={"ID":"1bb630b3-a283-4fa8-9887-1202daceae1a","Type":"ContainerStarted","Data":"4f4a1bfc1330405cae2c687bf7060c46895ba1eab841a1d5cb3f79c0c34b67cf"} Nov 03 13:03:55.827280 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:55.827175 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" podStartSLOduration=6.827154709 podStartE2EDuration="6.827154709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:54.936541975 +0000 UTC m=+7.753633041" watchObservedRunningTime="2026-11-03 13:03:55.827154709 +0000 UTC m=+8.644245776" Nov 03 13:03:55.937349 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:55.937282 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pjnlk" podStartSLOduration=3.286489339 podStartE2EDuration="7.937261945s" totalImagesPullingTime="4.650772606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:55.828453184 +0000 UTC m=+8.645544250" watchObservedRunningTime="2026-11-03 13:03:55.937261945 +0000 UTC m=+8.754353012" Nov 03 13:03:56.731232 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:56.731196 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" event={"ID":"d86486e7-2eb7-4200-a35b-8025870fc018","Type":"ContainerStarted","Data":"02ce808c95a36f21681729413d345893011c279751b626d8baeb75f60d6e9788"} Nov 03 13:03:57.305523 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:57.305482 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:57.305690 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:57.305573 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:57.305746 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.305736 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:57.305823 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.305813 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:04:05.305795417 +0000 UTC m=+18.122886489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:03:57.305900 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.305882 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:57.305957 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.305914 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret podName:89e41adf-23e2-4eaa-a335-59002579bdce nodeName:}" failed. No retries permitted until 2026-11-03 13:04:05.3059032 +0000 UTC m=+18.122994245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret") pod "global-pull-secret-syncer-ncrq2" (UID: "89e41adf-23e2-4eaa-a335-59002579bdce") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:03:57.406632 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:57.406593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:57.406840 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.406763 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:03:57.406907 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.406877 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:03:57.406907 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.406892 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nmhgn for pod openshift-network-diagnostics/network-check-target-hfbh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:57.407016 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.406974 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn podName:a7dc614f-3d12-4003-b572-8dbcbfcf7b6b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:05.406954527 +0000 UTC m=+18.224045572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmhgn" (UniqueName: "kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn") pod "network-check-target-hfbh8" (UID: "a7dc614f-3d12-4003-b572-8dbcbfcf7b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:03:57.649977 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:57.649949 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:57.650144 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.650066 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:03:57.650457 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:57.650435 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:57.650567 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.650546 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:03:57.650652 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:57.650600 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:57.650710 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:57.650685 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:03:58.604040 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:58.604011 2585 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 13:03:58.739888 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:58.739852 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" event={"ID":"d86486e7-2eb7-4200-a35b-8025870fc018","Type":"ContainerStarted","Data":"4a5b5d6ad53343e56d6f165ced267734cdd3e184165284de5a67800a999d9ca5"} Nov 03 13:03:58.744658 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:58.744628 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5zlvj" event={"ID":"48d92b6c-4577-458c-b6a7-7b11300ca6b1","Type":"ContainerStarted","Data":"9e011274234a8414efa176dd49fa192112ef5550b2edac1c23c13416628eff31"} Nov 03 13:03:58.747082 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:58.747052 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j99cl" event={"ID":"8ed56df7-b752-4f2d-b941-0b7df5aa6df9","Type":"ContainerStarted","Data":"0b85afa91f89db717a58d19ac6bef21d0b6ddf64ab0b568949b29955d82aee97"} Nov 03 13:03:58.852229 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:58.852124 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tgrzf" podStartSLOduration=5.274941105 podStartE2EDuration="10.852104845s" totalImagesPullingTime="5.57716374s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:55.937574042 +0000 UTC m=+8.754665109" watchObservedRunningTime="2026-11-03 13:03:58.852104845 +0000 UTC m=+11.669195912" Nov 03 13:03:58.962813 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:58.962739 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5zlvj" podStartSLOduration=2.470213368 podStartE2EDuration="10.962718975s" totalImagesPullingTime="8.492505607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:58.852512532 +0000 UTC m=+11.669603603" watchObservedRunningTime="2026-11-03 13:03:58.962718975 +0000 UTC m=+11.779810043" Nov 03 13:03:58.963333 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:58.963295 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-j99cl" podStartSLOduration=2.66166855 podStartE2EDuration="10.963250327s" totalImagesPullingTime="8.301581777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:03:58.962471685 +0000 UTC m=+11.779562756" watchObservedRunningTime="2026-11-03 13:03:58.963250327 +0000 UTC m=+11.780341395" Nov 03 13:03:59.576049 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:59.575900 2585 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:03:58.604038074Z","UUID":"91b6ca3b-d4bc-494c-85cd-bd9a862521d7","Handler":null,"Name":"","Endpoint":""} Nov 03 13:03:59.577499 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:59.577466 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 13:03:59.577499 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:59.577495 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 13:03:59.645974 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:59.645940 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:03:59.645974 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:59.645964 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:03:59.646490 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:59.646092 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:03:59.646490 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:59.646108 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:03:59.646490 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:59.646203 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:03:59.646490 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:03:59.646264 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:03:59.753357 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:59.753275 2585 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="9ff868439c3c389c3405bf274ad3e45028fd1d9066147122b86de845227df8db" exitCode=0 Nov 03 13:03:59.753520 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:03:59.753350 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" event={"ID":"ba4c7dc1-2eea-456e-bc9a-b6d02814657f","Type":"ContainerDied","Data":"9ff868439c3c389c3405bf274ad3e45028fd1d9066147122b86de845227df8db"} Nov 03 13:04:00.684094 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:00.683895 2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:04:00.684499 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:00.684479 2585 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:04:00.756879 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:00.756838 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" event={"ID":"8b2a6fcd-a217-4a03-994a-9261ed191c64","Type":"ContainerStarted","Data":"6b05b1c32c18c14925fe923a0f11e8ab15ae152060917acb7b38d96ac73aeb77"} Nov 03 13:04:00.759324 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:00.759297 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" event={"ID":"d86486e7-2eb7-4200-a35b-8025870fc018","Type":"ContainerStarted","Data":"fb295f47a042581e97afdb9909dba31d743cd63be0d0567b2e0f7054d64ac781"} Nov 03 13:04:00.761304 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:00.761273 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k8dl6" event={"ID":"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c","Type":"ContainerStarted","Data":"5301a0550b088c3e2f00ccf2a6d6eeb15bc9a522379e436d2776f7ecb7839ea7"} Nov 03 13:04:00.901765 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:00.901709 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k8dl6" podStartSLOduration=4.18800107 podStartE2EDuration="12.901695853s" totalImagesPullingTime="8.713694783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:00.90142912 +0000 UTC m=+13.718520188" watchObservedRunningTime="2026-11-03 13:04:00.901695853 +0000 UTC m=+13.718786918" Nov 03 13:04:01.009951 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:01.009858 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" podStartSLOduration=3.083466896 podStartE2EDuration="13.00984173s" totalImagesPullingTime="9.926374834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:01.009262202 +0000 UTC m=+13.826353269" watchObservedRunningTime="2026-11-03 13:04:01.00984173 +0000 UTC m=+13.826932796" Nov 03 13:04:01.117601 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:01.117545 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" podStartSLOduration=2.406074817 podStartE2EDuration="13.117530203s" totalImagesPullingTime="10.711455386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:01.117086985 +0000 UTC m=+13.934178052" watchObservedRunningTime="2026-11-03 13:04:01.117530203 +0000 UTC m=+13.934621281" Nov 03 13:04:01.645482 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:01.645405 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:01.645482 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:01.645421 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:01.645482 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:01.645466 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:01.645751 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:01.645570 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:01.645751 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:01.645685 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:01.645868 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:01.645800 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:01.765698 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:01.765668 2585 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="c000d2841716b42e04539f13ee736d74c46fb1c76465cae073e066c25329ef26" exitCode=0 Nov 03 13:04:01.766270 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:01.765932 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" event={"ID":"ba4c7dc1-2eea-456e-bc9a-b6d02814657f","Type":"ContainerDied","Data":"c000d2841716b42e04539f13ee736d74c46fb1c76465cae073e066c25329ef26"} Nov 03 13:04:02.156902 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:02.156863 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:04:02.157065 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:02.157038 2585 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:04:02.157531 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:02.157503 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:04:03.645368 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:03.645318 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:03.646038 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:03.645391 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:03.646038 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:03.645408 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:03.646038 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:03.645510 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:03.646038 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:03.645576 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:03.646038 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:03.645658 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:03.772445 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:03.772415 2585 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="dd672ea77edd3418dd81027195e6a7b98e1aa245fe474e986553476620d03a8a" exitCode=0 Nov 03 13:04:03.772605 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:03.772460 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" event={"ID":"ba4c7dc1-2eea-456e-bc9a-b6d02814657f","Type":"ContainerDied","Data":"dd672ea77edd3418dd81027195e6a7b98e1aa245fe474e986553476620d03a8a"} Nov 03 13:04:05.370563 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.370304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:05.371050 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.370614 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:05.371050 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.370476 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:05.371050 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.370704 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:04:21.37068039 +0000 UTC m=+34.187771442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:05.371050 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.370735 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:05.371050 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.370808 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret podName:89e41adf-23e2-4eaa-a335-59002579bdce nodeName:}" failed. No retries permitted until 2026-11-03 13:04:21.370791716 +0000 UTC m=+34.187882771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret") pod "global-pull-secret-syncer-ncrq2" (UID: "89e41adf-23e2-4eaa-a335-59002579bdce") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:05.471389 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.471349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:05.471565 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.471474 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:04:05.471565 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.471489 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:04:05.471565 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.471498 2585 projected.go:196] Error preparing data for projected volume kube-api-access-nmhgn for pod openshift-network-diagnostics/network-check-target-hfbh8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:05.471565 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.471553 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn podName:a7dc614f-3d12-4003-b572-8dbcbfcf7b6b nodeName:}" failed. No retries permitted until 2026-11-03 13:04:21.47153635 +0000 UTC m=+34.288627408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nmhgn" (UniqueName: "kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn") pod "network-check-target-hfbh8" (UID: "a7dc614f-3d12-4003-b572-8dbcbfcf7b6b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:05.644998 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.644794 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:05.645157 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.644794 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:05.645157 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.645092 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:05.645278 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.644821 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:05.645278 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.645193 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:05.645278 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:05.645239 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:05.779899 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.779824 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:04:05.780199 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.780175 2585 generic.go:356] "Generic (PLEG): container finished" podID="f9f28189-a808-48a4-b2da-d1950b84c7d0" containerID="bda8a73558cd0b85ca66dc4da57916070357bf07ab0a7fed38175cfe182ea03d" exitCode=1 Nov 03 13:04:05.780258 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.780218 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerStarted","Data":"b5b497a5fc49c2095bbebffcb270fb2b0898b7958a816fe3f9a564687356aecb"} Nov 03 13:04:05.780312 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.780274 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerStarted","Data":"9a106de700d4a4f546cd8ffe85bb48b66123ec636de4e6dd5c76f824a03b100a"} Nov 03 13:04:05.780380 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.780322 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerStarted","Data":"1725d817df7be0e63383a923ecfcd12cb261a42071c8672f5ece572781ec1f63"} Nov 03 13:04:05.780380 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.780370 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerStarted","Data":"6a9720788876633f8d930e1bcea6c22829d227e052a7e45bce3041f895aeb6a4"} Nov 03 13:04:05.780473 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.780413 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerDied","Data":"bda8a73558cd0b85ca66dc4da57916070357bf07ab0a7fed38175cfe182ea03d"} Nov 03 13:04:05.780473 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:05.780445 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerStarted","Data":"b72dc2f54a6f1d3f0ac0349db17d5031039cdffec2f3065a2efa405a85939423"} Nov 03 13:04:06.785268 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:06.785234 2585 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="c8eb5381b01ed50139ec168d70e142d350ea16d55977c1227b41dc4a3951a265" exitCode=0 Nov 03 13:04:06.785733 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:06.785279 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" event={"ID":"ba4c7dc1-2eea-456e-bc9a-b6d02814657f","Type":"ContainerDied","Data":"c8eb5381b01ed50139ec168d70e142d350ea16d55977c1227b41dc4a3951a265"} Nov 03 13:04:07.645945 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:07.645857 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:07.646113 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:07.645965 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:07.646113 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:07.645993 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:07.646113 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:07.645965 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:07.646113 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:07.646068 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:07.646303 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:07.646153 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:07.790135 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:07.790106 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:04:07.790553 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:07.790458 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerStarted","Data":"cf05189a1f1164a8cdfd0d441ec94f792e1823e7bbc6cadd8852c8e4c1ca13bf"} Nov 03 13:04:07.793073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:07.793051 2585 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="0caef70fcfd9b913a4ef099b4bdc51c451280bb77ad10eb4f902706ecf9df993" exitCode=0 Nov 03 13:04:07.793188 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:07.793082 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" event={"ID":"ba4c7dc1-2eea-456e-bc9a-b6d02814657f","Type":"ContainerDied","Data":"0caef70fcfd9b913a4ef099b4bdc51c451280bb77ad10eb4f902706ecf9df993"} Nov 03 13:04:08.798227 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:08.798194 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" event={"ID":"ba4c7dc1-2eea-456e-bc9a-b6d02814657f","Type":"ContainerStarted","Data":"381cd010e7ddbe52292a1a99c240bde50659a37cc73ff27886f10d6c1739a150"} Nov 03 13:04:08.958506 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:08.958455 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qm2ff" podStartSLOduration=8.059907301 podStartE2EDuration="20.958442003s" totalImagesPullingTime="12.898534702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:08.958119988 +0000 UTC m=+21.775211054" watchObservedRunningTime="2026-11-03 13:04:08.958442003 +0000 UTC m=+21.775533099" Nov 03 13:04:09.644500 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:09.644464 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:09.644500 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:09.644488 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:09.644800 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:09.644466 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:09.644800 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:09.644586 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:09.644800 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:09.644644 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:09.644800 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:09.644699 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:09.803458 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:09.803383 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:04:09.804114 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:09.803709 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerStarted","Data":"d1081534aa09146f131e21f6bdef3c99d67fa8cc13c1b8b4249ff922b37932f9"} Nov 03 13:04:09.804209 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:09.804193 2585 scope.go:122] "RemoveContainer" containerID="bda8a73558cd0b85ca66dc4da57916070357bf07ab0a7fed38175cfe182ea03d" Nov 03 13:04:09.804350 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:09.804331 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:10.810160 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:10.810131 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:04:10.810573 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:10.810507 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" event={"ID":"f9f28189-a808-48a4-b2da-d1950b84c7d0","Type":"ContainerStarted","Data":"67248a0ad3b5d5dc0664f226b62394083cce11aedcac26d8424217b40e1861b3"} Nov 03 13:04:10.810842 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:10.810812 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:10.810952 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:10.810897 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:10.825349 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:10.825329 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:10.829208 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:10.829192 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:11.066454 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:11.066349 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" podStartSLOduration=8.146861423 podStartE2EDuration="23.066334297s" totalImagesPullingTime="14.919472874s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:11.048831409 +0000 UTC m=+23.865922475" watchObservedRunningTime="2026-11-03 13:04:11.066334297 +0000 UTC m=+23.883425363" Nov 03 13:04:11.644760 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:11.644730 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:11.645002 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:11.644741 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:11.645002 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:11.644875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:11.645002 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:11.644870 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:11.645002 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:11.644959 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:11.645167 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:11.645016 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:12.025996 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:12.025964 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncrq2"] Nov 03 13:04:12.026542 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:12.026192 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:12.026542 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:12.026310 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:12.028925 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:12.028886 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hfbh8"] Nov 03 13:04:12.029060 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:12.029006 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:12.029117 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:12.029100 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:12.029613 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:12.029590 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r8f65"] Nov 03 13:04:12.029718 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:12.029706 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:12.029857 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:12.029836 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:13.645117 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:13.645076 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:13.645548 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:13.645082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:13.645548 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:13.645196 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:13.645548 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:13.645289 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:13.645548 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:13.645082 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:13.645548 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:13.645396 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:15.644885 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:15.644851 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:15.645351 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:15.644892 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:15.645351 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:15.644862 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:15.645351 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:15.644970 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:15.645351 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:15.645038 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:15.645351 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:15.645133 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:17.645979 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:17.645796 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:17.646349 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:17.645846 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:17.646349 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:17.646048 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r8f65" podUID="cf9f8413-6613-4db2-9dd1-20742c2af7ed" Nov 03 13:04:17.646349 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:17.646121 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ncrq2" podUID="89e41adf-23e2-4eaa-a335-59002579bdce" Nov 03 13:04:17.646349 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:17.645874 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:17.646349 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:17.646188 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hfbh8" podUID="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" Nov 03 13:04:18.031156 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.031130 2585 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeReady" Nov 03 13:04:18.031329 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.031260 2585 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 13:04:18.139836 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.139744 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b5n5d"] Nov 03 13:04:18.141750 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.141728 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.186036 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.185998 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pnc5z"] Nov 03 13:04:18.187889 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.187871 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b5n5d"] Nov 03 13:04:18.187988 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.187902 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pnc5z"] Nov 03 13:04:18.188023 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.187987 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:18.193290 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.193255 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 13:04:18.193401 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.193360 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m8wc5\"" Nov 03 13:04:18.193561 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.193543 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 13:04:18.236567 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.236529 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 13:04:18.236751 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.236616 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 13:04:18.236751 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.236696 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 13:04:18.236751 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.236705 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9jvp2\"" Nov 03 13:04:18.261764 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.261732 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7824f\" (UniqueName: \"kubernetes.io/projected/0c6e919c-8135-4ac2-b2fd-aafb531180be-kube-api-access-7824f\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.261932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.261795 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6e919c-8135-4ac2-b2fd-aafb531180be-tmp-dir\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.261932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.261862 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.261932 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.261916 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-cert\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:18.262061 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.261991 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvf9\" (UniqueName: \"kubernetes.io/projected/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-kube-api-access-dhvf9\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:18.262061 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.262021 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6e919c-8135-4ac2-b2fd-aafb531180be-config-volume\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.362646 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.362608 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-cert\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:18.362864 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.362675 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvf9\" (UniqueName: \"kubernetes.io/projected/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-kube-api-access-dhvf9\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:18.362864 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.362701 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6e919c-8135-4ac2-b2fd-aafb531180be-config-volume\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.362864 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.362722 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7824f\" (UniqueName: \"kubernetes.io/projected/0c6e919c-8135-4ac2-b2fd-aafb531180be-kube-api-access-7824f\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.362864 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.362753 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6e919c-8135-4ac2-b2fd-aafb531180be-tmp-dir\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.362864 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.362789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.363096 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:18.362887 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:18.363096 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:18.362948 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls podName:0c6e919c-8135-4ac2-b2fd-aafb531180be nodeName:}" failed. No retries permitted until 2026-11-03 13:04:18.862929053 +0000 UTC m=+31.680020098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls") pod "dns-default-b5n5d" (UID: "0c6e919c-8135-4ac2-b2fd-aafb531180be") : secret "dns-default-metrics-tls" not found Nov 03 13:04:18.363352 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.363333 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6e919c-8135-4ac2-b2fd-aafb531180be-tmp-dir\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.363509 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.363489 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6e919c-8135-4ac2-b2fd-aafb531180be-config-volume\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.366921 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.366897 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-cert\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:18.418150 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.418115 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7824f\" (UniqueName: \"kubernetes.io/projected/0c6e919c-8135-4ac2-b2fd-aafb531180be-kube-api-access-7824f\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.418309 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.418290 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvf9\" (UniqueName: \"kubernetes.io/projected/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-kube-api-access-dhvf9\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:18.497896 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.497864 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:18.866039 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.865909 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pnc5z"] Nov 03 13:04:18.866477 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:18.866388 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:18.866547 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:18.866499 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:18.866601 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:18.866554 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls podName:0c6e919c-8135-4ac2-b2fd-aafb531180be nodeName:}" failed. No retries permitted until 2026-11-03 13:04:19.866537686 +0000 UTC m=+32.683628730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls") pod "dns-default-b5n5d" (UID: "0c6e919c-8135-4ac2-b2fd-aafb531180be") : secret "dns-default-metrics-tls" not found Nov 03 13:04:18.870470 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:04:18.870446 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab5990d_5210_417d_80f9_d5d8bb0f7e03.slice/crio-723454fad717352ff4bf75bd39271d9f6143bf7379eb93c9e65534d7999cd976 WatchSource:0}: Error finding container 723454fad717352ff4bf75bd39271d9f6143bf7379eb93c9e65534d7999cd976: Status 404 returned error can't find the container with id 723454fad717352ff4bf75bd39271d9f6143bf7379eb93c9e65534d7999cd976 Nov 03 13:04:19.645005 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.644969 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:19.645184 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.645102 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:19.645184 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.644981 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:19.694313 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.694275 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 13:04:19.694313 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.694312 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nbm\"" Nov 03 13:04:19.694503 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.694313 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 13:04:19.741016 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.740986 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 13:04:19.741016 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.741007 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnhgg\"" Nov 03 13:04:19.741179 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.741077 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 13:04:19.741179 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.741150 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pvm6f\"" Nov 03 13:04:19.831481 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.831448 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pnc5z" event={"ID":"3ab5990d-5210-417d-80f9-d5d8bb0f7e03","Type":"ContainerStarted","Data":"723454fad717352ff4bf75bd39271d9f6143bf7379eb93c9e65534d7999cd976"} Nov 03 13:04:19.872870 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:19.872835 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:19.873227 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:19.872987 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:19.873227 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:19.873058 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls podName:0c6e919c-8135-4ac2-b2fd-aafb531180be nodeName:}" failed. No retries permitted until 2026-11-03 13:04:21.87304243 +0000 UTC m=+34.690133474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls") pod "dns-default-b5n5d" (UID: "0c6e919c-8135-4ac2-b2fd-aafb531180be") : secret "dns-default-metrics-tls" not found Nov 03 13:04:21.382233 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.382196 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:21.382727 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:21.382357 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:21.382727 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.382390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:21.382727 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:21.382433 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:04:53.382412989 +0000 UTC m=+66.199504035 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : secret "metrics-daemon-secret" not found Nov 03 13:04:21.384888 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.384864 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:21.456502 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.456464 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:21.483718 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.483269 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:21.485852 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.485815 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:21.762096 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.762061 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:21.821530 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.820995 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ncrq2"] Nov 03 13:04:21.827634 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:04:21.827598 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e41adf_23e2_4eaa_a335_59002579bdce.slice/crio-fae2b865f46285a4c89708abf1924079adeb19e40f28370184b5a97935427b85 WatchSource:0}: Error finding container fae2b865f46285a4c89708abf1924079adeb19e40f28370184b5a97935427b85: Status 404 returned error can't find the container with id fae2b865f46285a4c89708abf1924079adeb19e40f28370184b5a97935427b85 Nov 03 13:04:21.836626 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.836597 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ncrq2" event={"ID":"89e41adf-23e2-4eaa-a335-59002579bdce","Type":"ContainerStarted","Data":"fae2b865f46285a4c89708abf1924079adeb19e40f28370184b5a97935427b85"} Nov 03 13:04:21.885727 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:21.885705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:21.885860 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:21.885845 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:21.885911 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:21.885902 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls podName:0c6e919c-8135-4ac2-b2fd-aafb531180be nodeName:}" failed. No retries permitted until 2026-11-03 13:04:25.88588473 +0000 UTC m=+38.702975788 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls") pod "dns-default-b5n5d" (UID: "0c6e919c-8135-4ac2-b2fd-aafb531180be") : secret "dns-default-metrics-tls" not found Nov 03 13:04:22.129019 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:22.128985 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hfbh8"] Nov 03 13:04:22.131954 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:04:22.131927 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7dc614f_3d12_4003_b572_8dbcbfcf7b6b.slice/crio-d8dd570f958fea83f1a71365ca5d691a1c681a593e138568b734ef95b893a24d WatchSource:0}: Error finding container d8dd570f958fea83f1a71365ca5d691a1c681a593e138568b734ef95b893a24d: Status 404 returned error can't find the container with id d8dd570f958fea83f1a71365ca5d691a1c681a593e138568b734ef95b893a24d Nov 03 13:04:22.839950 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:22.839914 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pnc5z" event={"ID":"3ab5990d-5210-417d-80f9-d5d8bb0f7e03","Type":"ContainerStarted","Data":"a5a4243bdd2929864127b05e69b0c8fc23a0e383b60175d729b70bdadf1c7152"} Nov 03 13:04:22.840999 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:22.840978 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hfbh8" event={"ID":"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b","Type":"ContainerStarted","Data":"d8dd570f958fea83f1a71365ca5d691a1c681a593e138568b734ef95b893a24d"} Nov 03 13:04:22.946065 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:22.945968 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pnc5z" podStartSLOduration=1.958539281 podStartE2EDuration="4.945944618s" totalImagesPullingTime="2.987405337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:22.945008166 +0000 UTC m=+35.762099232" watchObservedRunningTime="2026-11-03 13:04:22.945944618 +0000 UTC m=+35.763035685" Nov 03 13:04:23.182460 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.182423 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pwsr8"] Nov 03 13:04:23.197132 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.197099 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.238991 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.238959 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pwsr8"] Nov 03 13:04:23.259811 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.259781 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 13:04:23.259811 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.259786 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 13:04:23.260032 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.259785 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 13:04:23.260032 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.259853 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pvzn\"" Nov 03 13:04:23.260032 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.259956 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 13:04:23.294654 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.294481 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-crio-socket\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.294654 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.294532 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6jcz\" (UniqueName: \"kubernetes.io/projected/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-api-access-s6jcz\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.294654 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.294574 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-data-volume\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.294654 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.294605 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.294654 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.294631 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.395319 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.395281 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-crio-socket\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.395594 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.395331 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jcz\" (UniqueName: \"kubernetes.io/projected/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-api-access-s6jcz\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.395594 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.395372 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-data-volume\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.395594 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.395411 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.395594 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.395437 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.395594 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:23.395580 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:23.395872 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:23.395657 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.895637406 +0000 UTC m=+36.712728466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:23.395924 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:23.395902 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:04:23.395990 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:23.395979 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.895963166 +0000 UTC m=+36.713054224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : configmap "kube-rbac-proxy" not found Nov 03 13:04:23.396098 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.396067 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-crio-socket\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.396223 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.396126 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-data-volume\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.419150 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.419116 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-9xmlk-2g7w7"] Nov 03 13:04:23.439496 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.439425 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.451041 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.451016 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jcz\" (UniqueName: \"kubernetes.io/projected/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-api-access-s6jcz\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.475794 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.475746 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9xmlk-2g7w7"] Nov 03 13:04:23.488951 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.488888 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 13:04:23.489163 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.488963 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-697z8\"" Nov 03 13:04:23.540216 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.540167 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 13:04:23.597056 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.597021 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.597230 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.597097 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.597230 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.597137 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5l9h\" (UniqueName: \"kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.597230 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.597178 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.698254 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.698169 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.698417 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.698258 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.698417 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.698290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5l9h\" (UniqueName: \"kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.698417 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.698319 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.698782 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.698752 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.698899 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.698845 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.699233 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.699210 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.760646 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.760613 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5l9h\" (UniqueName: \"kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.766241 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.766214 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:23.900228 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.900199 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.900836 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:23.900237 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:23.900836 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:23.900326 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:04:23.900836 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:23.900389 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:23.900836 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:23.900393 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:24.900373514 +0000 UTC m=+37.717464566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : configmap "kube-rbac-proxy" not found Nov 03 13:04:23.900836 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:23.900456 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:24.900439342 +0000 UTC m=+37.717530393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:24.159281 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.159226 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-9xmlk-2g7w7"] Nov 03 13:04:24.163848 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:04:24.163707 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31d9405_aac0_42bd_8c9a_977b6328bc65.slice/crio-81a0b663aae2a9e50f52cef108bf7d7f29c256ce7a85f067f346918dcd326d0b WatchSource:0}: Error finding container 81a0b663aae2a9e50f52cef108bf7d7f29c256ce7a85f067f346918dcd326d0b: Status 404 returned error can't find the container with id 81a0b663aae2a9e50f52cef108bf7d7f29c256ce7a85f067f346918dcd326d0b Nov 03 13:04:24.356512 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.356471 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk"] Nov 03 13:04:24.376447 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.376405 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" Nov 03 13:04:24.415092 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.415004 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk"] Nov 03 13:04:24.425378 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.425347 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8p2l7\"" Nov 03 13:04:24.441602 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.441571 2585 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-29qhq"] Nov 03 13:04:24.448518 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.448490 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.497869 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.497832 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-29qhq"] Nov 03 13:04:24.503815 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.503747 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 13:04:24.503962 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.503902 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 13:04:24.503962 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.503930 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 13:04:24.504069 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.503997 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:24.504069 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.504054 2585 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 13:04:24.505406 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.505382 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rckcg\" (UniqueName: \"kubernetes.io/projected/b0d94b94-63a5-404a-b169-e120292fcbe2-kube-api-access-rckcg\") pod \"migrator-66df5d496b-2rzqk\" (UID: \"b0d94b94-63a5-404a-b169-e120292fcbe2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" Nov 03 13:04:24.526131 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.526091 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xbj7f\"" Nov 03 13:04:24.526131 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.526093 2585 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 13:04:24.606437 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.606398 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-cabundle\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.606619 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.606486 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlnwn\" (UniqueName: \"kubernetes.io/projected/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-kube-api-access-qlnwn\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.606619 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.606572 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckcg\" (UniqueName: \"kubernetes.io/projected/b0d94b94-63a5-404a-b169-e120292fcbe2-kube-api-access-rckcg\") pod \"migrator-66df5d496b-2rzqk\" (UID: \"b0d94b94-63a5-404a-b169-e120292fcbe2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" Nov 03 13:04:24.606619 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.606609 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-key\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.661169 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.661128 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckcg\" (UniqueName: \"kubernetes.io/projected/b0d94b94-63a5-404a-b169-e120292fcbe2-kube-api-access-rckcg\") pod \"migrator-66df5d496b-2rzqk\" (UID: \"b0d94b94-63a5-404a-b169-e120292fcbe2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" Nov 03 13:04:24.700977 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.700891 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" Nov 03 13:04:24.707246 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.707216 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-cabundle\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.707422 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.707308 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnwn\" (UniqueName: \"kubernetes.io/projected/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-kube-api-access-qlnwn\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.707422 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.707339 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-key\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.708073 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.708047 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-cabundle\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.710130 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.710101 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-key\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.763025 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.762970 2585 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnwn\" (UniqueName: \"kubernetes.io/projected/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-kube-api-access-qlnwn\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.775148 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.774933 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:24.857078 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.857045 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" event={"ID":"f31d9405-aac0-42bd-8c9a-977b6328bc65","Type":"ContainerStarted","Data":"81a0b663aae2a9e50f52cef108bf7d7f29c256ce7a85f067f346918dcd326d0b"} Nov 03 13:04:24.908880 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.908844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:24.908880 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:24.908885 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:24.909415 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:24.909055 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:24.909415 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:24.909125 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:26.909104223 +0000 UTC m=+39.726195292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:24.909514 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:24.909504 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:04:24.909557 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:24.909547 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:26.909533226 +0000 UTC m=+39.726624284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : configmap "kube-rbac-proxy" not found Nov 03 13:04:25.095254 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:25.095223 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk"] Nov 03 13:04:25.097884 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:04:25.097855 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d94b94_63a5_404a_b169_e120292fcbe2.slice/crio-ff9ec61283d188f20a5694524f2017b347e14c6fab419a6cc14eb7ca36ba6a85 WatchSource:0}: Error finding container ff9ec61283d188f20a5694524f2017b347e14c6fab419a6cc14eb7ca36ba6a85: Status 404 returned error can't find the container with id ff9ec61283d188f20a5694524f2017b347e14c6fab419a6cc14eb7ca36ba6a85 Nov 03 13:04:25.157831 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:25.157735 2585 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-29qhq"] Nov 03 13:04:25.161913 ip-10-0-132-174 kubenswrapper[2585]: W1103 13:04:25.161881 2585 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bc89ad_0df9_43d4_b142_9df5def7f3dd.slice/crio-ea9b3eba1954680db487803e50fcc6a634e44323d69678eab4407c302a3c4e7c WatchSource:0}: Error finding container ea9b3eba1954680db487803e50fcc6a634e44323d69678eab4407c302a3c4e7c: Status 404 returned error can't find the container with id ea9b3eba1954680db487803e50fcc6a634e44323d69678eab4407c302a3c4e7c Nov 03 13:04:25.862540 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:25.862459 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" event={"ID":"f4bc89ad-0df9-43d4-b142-9df5def7f3dd","Type":"ContainerStarted","Data":"ea9b3eba1954680db487803e50fcc6a634e44323d69678eab4407c302a3c4e7c"} Nov 03 13:04:25.864146 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:25.864100 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" event={"ID":"b0d94b94-63a5-404a-b169-e120292fcbe2","Type":"ContainerStarted","Data":"ff9ec61283d188f20a5694524f2017b347e14c6fab419a6cc14eb7ca36ba6a85"} Nov 03 13:04:25.917212 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:25.917098 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:25.917671 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:25.917270 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:25.917671 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:25.917357 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls podName:0c6e919c-8135-4ac2-b2fd-aafb531180be nodeName:}" failed. No retries permitted until 2026-11-03 13:04:33.917334278 +0000 UTC m=+46.734425334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls") pod "dns-default-b5n5d" (UID: "0c6e919c-8135-4ac2-b2fd-aafb531180be") : secret "dns-default-metrics-tls" not found Nov 03 13:04:26.870367 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:26.870289 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hfbh8" event={"ID":"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b","Type":"ContainerStarted","Data":"80819e51d939424a591c9a14e64fd71cde30b8af883739ad75f7d500f6a7b554"} Nov 03 13:04:26.871007 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:26.870987 2585 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:26.924648 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:26.924616 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:26.924648 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:26.924653 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:26.925091 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:26.924761 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:26.925091 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:26.924829 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:30.924811392 +0000 UTC m=+43.741902454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:26.925254 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:26.925172 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 13:04:26.925254 ip-10-0-132-174 kubenswrapper[2585]: E1103 13:04:26.925223 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:30.925208468 +0000 UTC m=+43.742299525 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : configmap "kube-rbac-proxy" not found Nov 03 13:04:26.977350 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:26.977290 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hfbh8" podStartSLOduration=34.690695744 podStartE2EDuration="38.977270664s" totalImagesPullingTime="4.28657492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:26.976523546 +0000 UTC m=+39.793614617" watchObservedRunningTime="2026-11-03 13:04:26.977270664 +0000 UTC m=+39.794361731" Nov 03 13:04:27.878862 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:27.878833 2585 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" event={"ID":"b0d94b94-63a5-404a-b169-e120292fcbe2","Type":"ContainerStarted","Data":"1dec3011cf41bf5ff54010afa2eb20ab5faa22d540238ee4ab9dcfebae0bfa00"} Nov 03 13:04:27.985672 ip-10-0-132-174 kubenswrapper[2585]: I1103 13:04:27.985609 2585 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" podStartSLOduration=1.5851379030000001 podStartE2EDuration="3.985592215s" totalImagesPullingTime="2.400454312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:27.984580022 +0000 UTC m=+40.801671091" watchObservedRunningTime="2026-11-03 13:04:27.985592215 +0000 UTC m=+40.802683272" Nov 03 13:04:28.610578 ip-10-0-132-174 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 13:04:28.629177 ip-10-0-132-174 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 13:04:28.629400 ip-10-0-132-174 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 13:04:28.629465 ip-10-0-132-174 systemd[1]: kubelet.service: Consumed 2.460s CPU time. Nov 03 13:04:28.640402 ip-10-0-132-174 systemd[1]: Starting Kubernetes Kubelet... Nov 03 13:04:28.775317 ip-10-0-132-174 kubenswrapper[4541]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:04:28.775317 ip-10-0-132-174 kubenswrapper[4541]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 13:04:28.775317 ip-10-0-132-174 kubenswrapper[4541]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:04:28.775317 ip-10-0-132-174 kubenswrapper[4541]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:04:28.777516 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777416 4541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:28.777516 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777503 4541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:28.777516 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777509 4541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:28.777516 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777513 4541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:28.777516 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777517 4541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:28.777516 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777521 4541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:28.777516 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777525 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777529 4541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777534 4541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777538 4541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777541 4541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777545 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777549 4541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777553 4541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777556 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777573 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777578 4541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777582 4541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777585 4541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777589 4541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777593 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777596 4541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777600 4541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777604 4541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777608 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:28.777966 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777612 4541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777616 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777621 4541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777625 4541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777629 4541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777634 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777638 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777642 4541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777646 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777650 4541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777654 4541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777658 4541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777662 4541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777667 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777671 4541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777675 4541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777680 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777684 4541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777688 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777693 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:28.778813 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777697 4541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777701 4541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777705 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777717 4541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777723 4541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777728 4541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777732 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777736 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777742 4541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777746 4541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777750 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777755 4541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777759 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777763 4541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777789 4541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777794 4541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777799 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777804 4541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777808 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777812 4541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:28.779678 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777817 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777821 4541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777826 4541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777830 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777838 4541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777845 4541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777850 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777855 4541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777859 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777863 4541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777867 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777872 4541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777876 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777881 4541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777885 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777890 4541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777902 4541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777907 4541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777911 4541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777916 4541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:28.780651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777920 4541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777925 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777929 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777933 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777937 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777942 4541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777946 4541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777951 4541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777955 4541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777961 4541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777967 4541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777974 4541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777980 4541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777985 4541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777989 4541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777994 4541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.777998 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.778003 4541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.778007 4541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:28.781510 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.778012 4541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.778016 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779466 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779478 4541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779484 4541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779489 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779494 4541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779500 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779504 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779508 4541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779520 4541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779525 4541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779529 4541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779534 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779538 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779543 4541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779547 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779551 4541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779556 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:28.782061 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779564 4541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779570 4541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779576 4541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779581 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779585 4541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779589 4541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779594 4541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779598 4541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779603 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779607 4541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779611 4541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779616 4541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779620 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779624 4541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779629 4541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779633 4541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779637 4541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779641 4541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779646 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779650 4541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:28.782547 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779655 4541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779659 4541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779663 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779667 4541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779679 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779684 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779688 4541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779692 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779700 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779704 4541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779708 4541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779713 4541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779717 4541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779721 4541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779725 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779730 4541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779734 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779738 4541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779744 4541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779748 4541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:28.783066 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779756 4541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779760 4541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779764 4541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779787 4541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779795 4541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779801 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779806 4541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779811 4541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779815 4541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779819 4541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779823 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779828 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779832 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779836 4541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779841 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779845 4541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779849 4541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779854 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779858 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:28.783572 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779863 4541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779867 4541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779871 4541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779876 4541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779880 4541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779884 4541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779888 4541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779893 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779897 4541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779901 4541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779905 4541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779909 4541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779913 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779920 4541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779924 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779928 4541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779933 4541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779937 4541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779941 4541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:28.784060 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779948 4541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779952 4541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779956 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779961 4541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779965 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779970 4541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779975 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779979 4541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779983 4541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779987 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.779992 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781933 4541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781947 4541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781951 4541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781955 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781958 4541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781960 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781964 4541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781967 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781969 4541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:28.784539 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781972 4541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781974 4541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781977 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781980 4541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781983 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781985 4541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781988 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781991 4541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.781995 4541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782000 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782004 4541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782008 4541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782011 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782014 4541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782017 4541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782019 4541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782022 4541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782026 4541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782031 4541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:28.785049 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782034 4541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782037 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782040 4541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782047 4541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782050 4541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782053 4541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782056 4541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782058 4541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782061 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782064 4541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782067 4541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782070 4541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782073 4541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782076 4541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782079 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782081 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782084 4541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782087 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782089 4541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782092 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:28.785521 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782094 4541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782097 4541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782100 4541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782102 4541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782106 4541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782108 4541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782111 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782114 4541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782116 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782119 4541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782122 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782127 4541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782130 4541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782132 4541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782135 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782138 4541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782141 4541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782143 4541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782146 4541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782149 4541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:28.786047 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782152 4541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782154 4541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782157 4541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782159 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782162 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782164 4541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782166 4541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782169 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782171 4541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782174 4541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782176 4541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782179 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782182 4541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782185 4541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782187 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782191 4541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782194 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782197 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782199 4541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782202 4541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:28.786534 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782204 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782207 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782209 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782217 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782220 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782222 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782226 4541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782228 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782231 4541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782233 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782236 4541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782239 4541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782244 4541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782248 4541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782251 4541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782254 4541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782256 4541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.782259 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782354 4541 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782362 4541 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 13:04:28.787089 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782368 4541 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782373 4541 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782378 4541 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782381 4541 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782385 4541 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782389 4541 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782393 4541 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782396 4541 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782403 4541 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782407 4541 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782410 4541 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782413 4541 flags.go:64] FLAG: --cgroup-root="" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782416 4541 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782419 4541 flags.go:64] FLAG: --client-ca-file="" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782422 4541 flags.go:64] FLAG: --cloud-provider="external" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782427 4541 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782432 4541 flags.go:64] FLAG: --cluster-domain="" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782435 4541 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782439 4541 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782442 4541 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782445 4541 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782449 4541 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782452 4541 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 13:04:28.787689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782456 4541 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782459 4541 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782462 4541 flags.go:64] FLAG: --contention-profiling="false" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782465 4541 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782468 4541 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782472 4541 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782475 4541 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782479 4541 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782482 4541 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782485 4541 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782488 4541 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782491 4541 flags.go:64] FLAG: --enable-server="true" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782495 4541 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782499 4541 flags.go:64] FLAG: --event-burst="100" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782502 4541 flags.go:64] FLAG: --event-qps="50" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782505 4541 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782508 4541 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782511 4541 flags.go:64] FLAG: --eviction-hard="" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782516 4541 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782519 4541 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782522 4541 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782525 4541 flags.go:64] FLAG: --eviction-soft="" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782528 4541 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782531 4541 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782536 4541 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 13:04:28.788280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782539 4541 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782542 4541 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782545 4541 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782548 4541 flags.go:64] FLAG: --feature-gates="" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782552 4541 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782555 4541 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782559 4541 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782562 4541 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782565 4541 flags.go:64] FLAG: --healthz-port="10248" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782569 4541 flags.go:64] FLAG: --help="false" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782571 4541 flags.go:64] FLAG: --hostname-override="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782575 4541 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782578 4541 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782581 4541 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782584 4541 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782587 4541 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782590 4541 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782593 4541 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782596 4541 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782599 4541 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782602 4541 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782605 4541 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782607 4541 flags.go:64] FLAG: --kube-reserved="" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782610 4541 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 13:04:28.788927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782614 4541 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782617 4541 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782620 4541 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782623 4541 flags.go:64] FLAG: --lock-file="" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782626 4541 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782629 4541 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782632 4541 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782639 4541 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782642 4541 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782645 4541 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782648 4541 flags.go:64] FLAG: --logging-format="text" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782651 4541 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782654 4541 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782657 4541 flags.go:64] FLAG: --manifest-url="" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782660 4541 flags.go:64] FLAG: --manifest-url-header="" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782665 4541 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782668 4541 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782672 4541 flags.go:64] FLAG: --max-pods="110" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782675 4541 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782678 4541 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782682 4541 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782685 4541 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782688 4541 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782691 4541 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782695 4541 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 13:04:28.789507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782701 4541 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782704 4541 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782707 4541 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782710 4541 flags.go:64] FLAG: --pod-cidr="" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782713 4541 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782716 4541 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782720 4541 flags.go:64] FLAG: --pods-per-core="0" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782723 4541 flags.go:64] FLAG: --port="10250" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782725 4541 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782728 4541 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e0a2f74ef738df33" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782732 4541 flags.go:64] FLAG: --qos-reserved="" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782736 4541 flags.go:64] FLAG: --read-only-port="10255" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782739 4541 flags.go:64] FLAG: --register-node="true" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782741 4541 flags.go:64] FLAG: --register-with-taints="" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782749 4541 flags.go:64] FLAG: --registry-burst="10" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782752 4541 flags.go:64] FLAG: --registry-qps="5" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782755 4541 flags.go:64] FLAG: --reserved-cpus="" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782758 4541 flags.go:64] FLAG: --reserved-memory="" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782762 4541 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782766 4541 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782781 4541 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782784 4541 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782787 4541 flags.go:64] FLAG: --runonce="false" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782790 4541 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782793 4541 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782797 4541 flags.go:64] FLAG: --seccomp-default="false" Nov 03 13:04:28.790201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782799 4541 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782803 4541 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782806 4541 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782809 4541 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782812 4541 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782815 4541 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782818 4541 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782821 4541 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782824 4541 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782827 4541 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782829 4541 flags.go:64] FLAG: --system-cgroups="" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782832 4541 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782837 4541 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782840 4541 flags.go:64] FLAG: --tls-cert-file="" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782843 4541 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782848 4541 flags.go:64] FLAG: --tls-min-version="" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782850 4541 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782854 4541 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782857 4541 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782860 4541 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782865 4541 flags.go:64] FLAG: --v="2" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782870 4541 flags.go:64] FLAG: --version="false" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782874 4541 flags.go:64] FLAG: --vmodule="" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782878 4541 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.782882 4541 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 13:04:28.790854 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783009 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783013 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783016 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783020 4541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783022 4541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783025 4541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783028 4541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783031 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783033 4541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783036 4541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783039 4541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783042 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783044 4541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783047 4541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783050 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783053 4541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783055 4541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783058 4541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783060 4541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:28.791500 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783064 4541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783068 4541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783072 4541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783075 4541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783078 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783081 4541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783084 4541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783087 4541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783093 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783096 4541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783099 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783101 4541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783104 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783107 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783110 4541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783113 4541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783115 4541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783118 4541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783121 4541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:28.791990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783123 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783126 4541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783129 4541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783131 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783134 4541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783136 4541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783139 4541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783141 4541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783144 4541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783147 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783149 4541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783152 4541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783155 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783159 4541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783163 4541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783167 4541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783170 4541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783173 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783175 4541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783178 4541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:28.792630 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783181 4541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783187 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783190 4541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783192 4541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783195 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783198 4541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783200 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783203 4541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783206 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783208 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783211 4541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783214 4541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783216 4541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783219 4541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783221 4541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783224 4541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783227 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783229 4541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783232 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:28.793273 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783234 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783237 4541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783240 4541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783242 4541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783245 4541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783247 4541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783250 4541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783253 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783255 4541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783258 4541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783260 4541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783263 4541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783265 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783268 4541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783273 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783276 4541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783279 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783282 4541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783284 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:28.793922 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783287 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783290 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783292 4541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783295 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783298 4541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783300 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783303 4541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783306 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783308 4541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783311 4541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.783337 4541 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783372 4541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783376 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783380 4541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783383 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783386 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:28.794390 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783388 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783391 4541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783394 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783397 4541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783399 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783402 4541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783404 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783407 4541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783410 4541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783412 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783415 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783418 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783423 4541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783426 4541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783428 4541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783431 4541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783433 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783436 4541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783438 4541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783441 4541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:28.794816 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783444 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783446 4541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783449 4541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783452 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783454 4541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783457 4541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783460 4541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783462 4541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783466 4541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783471 4541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783475 4541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783477 4541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783480 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783483 4541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783486 4541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783488 4541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783491 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783493 4541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783496 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:28.795573 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783499 4541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783501 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783504 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783506 4541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783511 4541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783515 4541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783518 4541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783520 4541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783523 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783526 4541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783528 4541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783531 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783533 4541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783536 4541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783539 4541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783541 4541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783544 4541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783547 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783549 4541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783552 4541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:28.796304 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783555 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783559 4541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783562 4541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783565 4541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783567 4541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783570 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783573 4541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783576 4541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783579 4541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783582 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783584 4541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783587 4541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783590 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783592 4541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783595 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783597 4541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783602 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783606 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783608 4541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:28.797007 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783611 4541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783614 4541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783617 4541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783619 4541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783622 4541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783624 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783627 4541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783630 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783632 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783635 4541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783637 4541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783640 4541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783643 4541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783645 4541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783648 4541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783650 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783653 4541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783656 4541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783659 4541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:28.797538 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783661 4541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783664 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783666 4541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.783669 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.787416 4541 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.787433 4541 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.787452 4541 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.787457 4541 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787512 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787518 4541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787522 4541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787526 4541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787529 4541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787532 4541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787535 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787537 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787540 4541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787543 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787545 4541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:28.798313 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787549 4541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787551 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787554 4541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787557 4541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787559 4541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787562 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787564 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787567 4541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787570 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787572 4541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787575 4541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787578 4541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787580 4541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787583 4541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787586 4541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787589 4541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787592 4541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787594 4541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787598 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787600 4541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:28.799205 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787603 4541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787606 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787608 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787611 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787613 4541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787616 4541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787618 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787621 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787625 4541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787630 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787633 4541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787636 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787639 4541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787642 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787644 4541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787647 4541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787649 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787653 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787656 4541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:28.800059 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787658 4541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787661 4541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787664 4541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787666 4541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787669 4541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787672 4541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787674 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787677 4541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787680 4541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787682 4541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787685 4541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787688 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787691 4541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787694 4541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787697 4541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787700 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787702 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787705 4541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787708 4541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787710 4541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:28.800735 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787713 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787716 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787719 4541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787722 4541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787725 4541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787727 4541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787730 4541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787733 4541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787736 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787738 4541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787742 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787745 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787747 4541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787750 4541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787753 4541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787756 4541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787760 4541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787763 4541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787766 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:28.801380 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787788 4541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787791 4541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787794 4541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787796 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787799 4541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787802 4541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787805 4541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787808 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787811 4541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787813 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787816 4541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787818 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787821 4541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787823 4541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787826 4541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787829 4541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:28.801866 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787831 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.787859 4541 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787994 4541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.787998 4541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788001 4541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788005 4541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788007 4541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788010 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788013 4541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788016 4541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788019 4541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788021 4541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788024 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788027 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788029 4541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788032 4541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:28.802294 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788034 4541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788037 4541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788040 4541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788042 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788045 4541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788047 4541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788050 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788053 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788056 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788058 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788061 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788063 4541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788066 4541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788069 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788072 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788074 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788077 4541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788080 4541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788083 4541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:28.802692 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788085 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788088 4541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788091 4541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788093 4541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788096 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788099 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788101 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788104 4541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788107 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788109 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788112 4541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788115 4541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788117 4541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788120 4541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788122 4541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788125 4541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788127 4541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788130 4541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788132 4541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788135 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:28.803169 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788137 4541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788141 4541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788143 4541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788146 4541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788148 4541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788151 4541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788154 4541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788156 4541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788159 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788162 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788164 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788167 4541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788169 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788172 4541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788175 4541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788177 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788180 4541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788182 4541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788185 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:28.803697 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788187 4541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788190 4541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788193 4541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788195 4541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788198 4541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788200 4541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788203 4541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788205 4541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788208 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788211 4541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788213 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788216 4541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788218 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788221 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788223 4541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788226 4541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788229 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788231 4541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788234 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:28.804236 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788236 4541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788239 4541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788249 4541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788253 4541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788257 4541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788260 4541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788262 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788265 4541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788268 4541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788270 4541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788273 4541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788276 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788279 4541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788282 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788286 4541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.788317 4541 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:04:28.804694 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788350 4541 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788354 4541 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788358 4541 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788361 4541 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788363 4541 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788366 4541 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788369 4541 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788372 4541 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788375 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788377 4541 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788380 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788383 4541 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788385 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788389 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788392 4541 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788395 4541 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788397 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788400 4541 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788403 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:28.805082 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788406 4541 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788409 4541 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788412 4541 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788416 4541 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788419 4541 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788422 4541 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788425 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788427 4541 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788430 4541 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788432 4541 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788435 4541 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788437 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788440 4541 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788442 4541 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788447 4541 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788451 4541 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788454 4541 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788457 4541 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788460 4541 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788463 4541 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:28.805583 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788465 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788468 4541 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788471 4541 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788474 4541 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788476 4541 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788479 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788482 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788485 4541 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788488 4541 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788490 4541 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788493 4541 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788495 4541 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788498 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788501 4541 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788503 4541 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788506 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788508 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788511 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788513 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788516 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:28.806081 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788519 4541 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788521 4541 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788524 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788526 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788529 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788532 4541 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788535 4541 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788537 4541 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788540 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788542 4541 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788545 4541 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788547 4541 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788550 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788552 4541 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788555 4541 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788557 4541 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788560 4541 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788562 4541 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788565 4541 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:28.806598 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788568 4541 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788571 4541 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788574 4541 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788576 4541 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788579 4541 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788582 4541 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788584 4541 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788587 4541 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788589 4541 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788593 4541 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788596 4541 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788598 4541 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788601 4541 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788603 4541 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788606 4541 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788608 4541 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788610 4541 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788613 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788616 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:28.807155 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788618 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788621 4541 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788623 4541 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788626 4541 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788630 4541 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788633 4541 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788635 4541 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788638 4541 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:28.788641 4541 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.788697 4541 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.789729 4541 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.789806 4541 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.789991 4541 server.go:1014] "Starting client certificate rotation" Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.790124 4541 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.790173 4541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 12:58:47 +0000 UTC" deadline="2028-07-31 06:26:52.660909106 +0000 UTC" Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.790206 4541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15257h22m23.870704363s" Nov 03 13:04:28.807679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.790421 4541 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:04:28.808142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.791549 4541 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:04:28.808142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.792909 4541 log.go:25] "Validated CRI v1 runtime API" Nov 03 13:04:28.808142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.794554 4541 log.go:25] "Validated CRI v1 image API" Nov 03 13:04:28.808142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.797132 4541 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 13:04:28.808142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.802181 4541 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a9751310-f22a-441f-9fed-e1ddc205e8fa:/dev/nvme0n1p4 f484694d-4008-4563-84bc-8dd7427d2bfc:/dev/nvme0n1p3] Nov 03 13:04:28.808310 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.802203 4541 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fb1aebcff1ab1345a58c682d1bdd3ea96b0ddb540f8a968e6ecdc67e9e21fa7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fb1aebcff1ab1345a58c682d1bdd3ea96b0ddb540f8a968e6ecdc67e9e21fa7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/268e0a0f4b5f525db7154cc7aaeb9342b395c96d17370c0d91cc03bccb8dd516/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/268e0a0f4b5f525db7154cc7aaeb9342b395c96d17370c0d91cc03bccb8dd516/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ba7aa05fc2b28f32027396cda92af8499002e51a3e9b85eb294808a56270fb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ba7aa05fc2b28f32027396cda92af8499002e51a3e9b85eb294808a56270fb8/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f4950b203312825d4741862e4ed5d84fbac74b94028dab9c710e68b8733ddad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f4950b203312825d4741862e4ed5d84fbac74b94028dab9c710e68b8733ddad/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/723454fad717352ff4bf75bd39271d9f6143bf7379eb93c9e65534d7999cd976/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/723454fad717352ff4bf75bd39271d9f6143bf7379eb93c9e65534d7999cd976/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72778de9f271e583a7afed12182177aa39d73fb9aed049cb862ba0d20210617e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72778de9f271e583a7afed12182177aa39d73fb9aed049cb862ba0d20210617e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8108f21a9c6fda01f5de299b4d89ec4742abc7f12f2850d8985c560bdb7b7665/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8108f21a9c6fda01f5de299b4d89ec4742abc7f12f2850d8985c560bdb7b7665/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81a0b663aae2a9e50f52cef108bf7d7f29c256ce7a85f067f346918dcd326d0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81a0b663aae2a9e50f52cef108bf7d7f29c256ce7a85f067f346918dcd326d0b/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caec9813de1596f33731e527d0e07328d7f9818c6240c19d6731d636a7b0f14b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caec9813de1596f33731e527d0e07328d7f9818c6240c19d6731d636a7b0f14b/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d45df744d7ce685c3686ce232a54f70ef8a7a0f294edb4d561ad8baaedc13944/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d45df744d7ce685c3686ce232a54f70ef8a7a0f294edb4d561ad8baaedc13944/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8dd570f958fea83f1a71365ca5d691a1c681a593e138568b734ef95b893a24d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8dd570f958fea83f1a71365ca5d691a1c681a593e138568b734ef95b893a24d/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6fd964a7cdff5d40e64edc030cace79a4941c44ee1931f68ac96bd8628141cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6fd964a7cdff5d40e64edc030cace79a4941c44ee1931f68ac96bd8628141cb/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea9b3eba1954680db487803e50fcc6a634e44323d69678eab4407c302a3c4e7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea9b3eba1954680db487803e50fcc6a634e44323d69678eab4407c302a3c4e7c/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fae2b865f46285a4c89708abf1924079adeb19e40f28370184b5a97935427b85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fae2b865f46285a4c89708abf1924079adeb19e40f28370184b5a97935427b85/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdad73a7eea44c8258a8c910b3198dbcc478f6aa231fc2ac9c9bb0b032b1f668/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdad73a7eea44c8258a8c910b3198dbcc478f6aa231fc2ac9c9bb0b032b1f668/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff9ec61283d188f20a5694524f2017b347e14c6fab419a6cc14eb7ca36ba6a85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff9ec61283d188f20a5694524f2017b347e14c6fab419a6cc14eb7ca36ba6a85/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0c6e919c-8135-4ac2-b2fd-aafb531180be/volumes/kubernetes.io~projected/kube-api-access-7824f:{mountpoint:/var/lib/kubelet/pods/0c6e919c-8135-4ac2-b2fd-aafb531180be/volumes/kubernetes.io~projected/kube-api-access-7824f major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bb630b3-a283-4fa8-9887-1202daceae1a/volumes/kubernetes.io~projected/kube-api-access-prpcz:{mountpoint:/var/lib/kubelet/pods/1bb630b3-a283-4fa8-9887-1202daceae1a/volumes/kubernetes.io~projected/kube-api-access-prpcz major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ab5990d-5210-417d-80f9-d5d8bb0f7e03/volumes/kubernetes.io~projected/kube-api-access-dhvf9:{mountpoint:/var/lib/kubelet/pods/3ab5990d-5210-417d-80f9-d5d8bb0f7e03/volumes/kubernetes.io~projected/kube-api-access-dhvf9 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ab5990d-5210-417d-80f9-d5d8bb0f7e03/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3ab5990d-5210-417d-80f9-d5d8bb0f7e03/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/48d92b6c-4577-458c-b6a7-7b11300ca6b1/volumes/kubernetes.io~projected/kube-api-access-hwzw9:{mountpoint:/var/lib/kubelet/pods/48d92b6c-4577-458c-b6a7-7b11300ca6b1/volumes/kubernetes.io~projected/kube-api-access-hwzw9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89e41adf-23e2-4eaa-a335-59002579bdce/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/89e41adf-23e2-4eaa-a335-59002579bdce/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c/volumes/kubernetes.io~projected/kube-api-access-7vwsq:{mountpoint:/var/lib/kubelet/pods/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c/volumes/kubernetes.io~projected/kube-api-access-7vwsq major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b2a6fcd-a217-4a03-994a-9261ed191c64/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8b2a6fcd-a217-4a03-994a-9261ed191c64/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b2a6fcd-a217-4a03-994a-9261ed191c64/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8b2a6fcd-a217-4a03-994a-9261ed191c64/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b2a6fcd-a217-4a03-994a-9261ed191c64/volumes/kubernetes.io~projected/kube-api-access-nnrpn:{mountpoint:/var/lib/kubelet/pods/8b2a6fcd-a217-4a03-994a-9261ed191c64/volumes/kubernetes.io~projected/kube-api-access-nnrpn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ed56df7-b752-4f2d-b941-0b7df5aa6df9/volumes/kubernetes.io~projected/kube-api-access-ks6h6:{mountpoint:/var/lib/kubelet/pods/8ed56df7-b752-4f2d-b941-0b7df5aa6df9/volumes/kubernetes.io~projected/kube-api-access-ks6h6 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b/volumes/kubernetes.io~projected/kube-api-access-nmhgn:{mountpoint:/var/lib/kubelet/pods/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b/volumes/kubernetes.io~projected/kube-api-access-nmhgn major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0d94b94-63a5-404a-b169-e120292fcbe2/volumes/kubernetes.io~projected/kube-api-access-rckcg:{mountpoint:/var/lib/kubelet/pods/b0d94b94-63a5-404a-b169-e120292fcbe2/volumes/kubernetes.io~projected/kube-api-access-rckcg major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba4c7dc1-2eea-456e-bc9a-b6d02814657f/volumes/kubernetes.io~projected/kube-api-access-zm6g2:{mountpoint:/var/lib/kubelet/pods/ba4c7dc1-2eea-456e-bc9a-b6d02814657f/volumes/kubernetes.io~projected/kube-api-access-zm6g2 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf9f8413-6613-4db2-9dd1-20742c2af7ed/volumes/kubernetes.io~projected/kube-api-access-qtzxj:{mountpoint:/var/lib/kubelet/pods/cf9f8413-6613-4db2-9dd1-20742c2af7ed/volumes/kubernetes.io~projected/kube-api-access-qtzxj major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d86486e7-2eb7-4200-a35b-8025870fc018/volumes/kubernetes.io~projected/kube-api-access-nnd4j:{mountpoint:/var/lib/kubelet/pods/d86486e7-2eb7-4200-a35b-8025870fc018/volumes/kubernetes.io~projected/kube-api-access-nnd4j major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055/volumes/kubernetes.io~projected/kube-api-access-s6jcz:{mountpoint:/var/lib/kubelet/pods/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055/volumes/kubernetes.io~projected/kube-api-access-s6jcz major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f31d9405-aac0-42bd-8c9a-977b6328bc65/volumes/kubernetes.io~projected/kube-api-access-q5l9h:{mountpoint:/var/lib/kubelet/pods/f31d9405-aac0-42bd-8c9a-977b6328bc65/volumes/kubernetes.io~projected/kube-api-access-q5l9h major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4bc89ad-0df9-43d4-b142-9df5def7f3dd/volumes/kubernetes.io~projected/kube-api-access-qlnwn:{mountpoint:/var/lib/kubelet/pods/f4bc89ad-0df9-43d4-b142-9df5def7f3dd/volumes/kubernetes.io~projected/kube-api-access-qlnwn major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4bc89ad-0df9-43d4-b142-9df5def7f3dd/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f4bc89ad-0df9-43d4-b142-9df5def7f3dd/volumes/kubernetes.io~secret/signing-key major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9f28189-a808-48a4-b2da-d1950b84c7d0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f9f28189-a808-48a4-b2da-d1950b84c7d0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9f28189-a808-48a4-b2da-d1950b84c7d0/volumes/kubernetes.io~projected/kube-api-access-mvswn:{mountpoint:/var/lib/kubelet/pods/f9f28189-a808-48a4-b2da-d1950b84c7d0/volumes/kubernetes.io~projected/kube-api-access-mvswn major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9f28189-a808-48a4-b2da-d1950b84c7d0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f9f28189-a808-48a4-b2da-d1950b84c7d0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5c62085a3ac1529462b1ef1951c9e7685a9539c0123cfa00498b2da9d2401511/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1002ba32f050f856ccda50014d6dc57659f7e338daaff7cf41d0438100021ba9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d3ff26ce7a4da4f36da83322072313c2a555067c5d61249de18c208c2ee3dfec/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/97cd55386405479a304565848e79d1952405e607658298accffbfc3231caae95/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b94c13027f5fdd744601f7810294a9ffd676c9a77345d171c630fdda7918fc1d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/13871d6e1f00a40ac8d72db81ae11ee4b5e77884f48767ec937cfcc6c819519c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/dc5cd06db557b47bff93a4ef1b2ba865d5f3680af3406e0b1124b55af9fa8fe6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/bf7a5e89d524dc669f49c8acf4f70ee3344967701fa693ceb907ee86ea876eef/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/dc98dde25351ed3855346cebbfbf40025272e2ef089c992093fc550f323b824c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/c17fa55a20e8558e29c6190eca7a7061bbd438642ec6fde11889eefe1db68ac4/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/3b17e76cef41138746791174ec1b51def8ae08ab22213b7031abce3604527603/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/0bb744a7012ffcbe190dd3bb114836e666bbeaadba3d6266dd999a144c95d6e3/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/64c35fd17d38d1e4c90c979096cb022775733301d234a6f5499ce4f67efffeb1/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8d2443d2b8fe6f772caf3e01351672a52489d5c3f60bb77ce067bdcb87f3f229/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/199d54e0f28afa0eb784e8ae20f4dafbc087fd20a429f24358128b246bee1de1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/330df17a45bfc6ca2519801c5ac870d09b79c281d0d351fceba83048b9d3dbf1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/3b4be87288fa8ea08f9dcecd516c9615a804d998f044026e04821a4cc43114f3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/aa5e1f4ed61dcd3abf5f5c5d50a50044c0c8e8dffa0fdfd16434a6d832a9e357/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/46e7e6d0e871a2e0b1032a80a690dde8d13cbd7ae6eb89e7d71ff8bda8266851/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0449c6a7697555f6881a2009183a81e0599bd12d2038f72d659915e239ea9665/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/9340a3f953f63312c646908d63d1a27a9857c5c7ba3c1874c78f269d688b3dbe/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/2f6587fdde266b9bb2efaa69bb68c427628f589ac449fa4ae53a7b839eef3711/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/209c7e8c7c2c8aab5c70df593bf8f78e6b9743edce1750b54ba691b88383f77a/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/0479c4b7b380a44c23cd72205c80a65b2c6d3041567c48f7f0cb2ad5948c0b1e/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/5435b7f152c536bf91bab311869ddfc416a4ead026b7264d39852ed44a91eb0f/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/cfbfeaa256b50970d2cbb314d0e815178a539e132552999c7432a78fbeac7a4b/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/ebd77cd048e38bb05dc547a7cca5da9b4080f6aebbe8baaff336f43367075c7e/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/2ccf89bea46ac272907e949c16a83719b7a2a7cfe6f9383015333ae9bd316a7a/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/50f098cc730c27f0956eed46bcf623be25c952ae3c0cd4927639bcf1d365c146/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d75c962e70c8a07ff4bd362e5ad9460fa21628b426044efa623f65a5ed0a15fe/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/4338e8e81214c5fdab286084f722c9b90efb50cee8a7a01d27b57bb23170c87d/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/e520a935527c56db546d79d7ae24e4dbc62d9629f3bc76480e5c5409a0242c4a/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/3bf369db25b7605154f64d0b747edbc70bb7a53721c4fc0312ddd4aa4885c5f2/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/df650e0ce1657a7e735d1399ae190312f11b255b9b251ad04ad27cf17bd3c507/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/98bef77073f4547cdf9704373df42b610aa6a688204376bba420800a32251018/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c94623ab9843a73b450987ab35bc547e91f1a0f015088d912897b66ae9da4e4d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/49a1957033c01cd77be84997bf3e764883ff19f19f5a8a5f10ce6e9cfbc8437d/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/49fc64ca836086d3061b8b7408204e859233cf835bd489b0bd9b02e122a5df36/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/faa0043fe3b75a359500d02d1176377dbf48ef329339c4625200d9d8586dd709/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/9bd22fb2c6d0ae8d07380edd26f9f54c40d25c42a6a663c22b947d415514b99d/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/000ae598f03837c3634e9e365615dda73a8754567e69de19ad7ab14e1c6546d3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/dd846489327d70b1f9038c9c5fd173c04a4ebc6c3660daee026e3837b54d1e54/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 13:04:28.809493 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.809231 4541 manager.go:217] Machine: {Timestamp:2026-11-03 13:04:28.808757813 +0000 UTC m=+0.122729665 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098427 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec210648c3ea5f491cff8be8b1aee627 SystemUUID:ec210648-c3ea-5f49-1cff-8be8b1aee627 BootID:8c76694b-7b45-4a7f-b919-8c9f990e729b Filesystems:[{Device:/var/lib/kubelet/pods/d86486e7-2eb7-4200-a35b-8025870fc018/volumes/kubernetes.io~projected/kube-api-access-nnd4j DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/268e0a0f4b5f525db7154cc7aaeb9342b395c96d17370c0d91cc03bccb8dd516/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f9f28189-a808-48a4-b2da-d1950b84c7d0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0c6e919c-8135-4ac2-b2fd-aafb531180be/volumes/kubernetes.io~projected/kube-api-access-7824f DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b/volumes/kubernetes.io~projected/kube-api-access-nmhgn DeviceMajor:0 DeviceMinor:209 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea9b3eba1954680db487803e50fcc6a634e44323d69678eab4407c302a3c4e7c/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/48d92b6c-4577-458c-b6a7-7b11300ca6b1/volumes/kubernetes.io~projected/kube-api-access-hwzw9 DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdad73a7eea44c8258a8c910b3198dbcc478f6aa231fc2ac9c9bb0b032b1f668/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b2a6fcd-a217-4a03-994a-9261ed191c64/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1bb630b3-a283-4fa8-9887-1202daceae1a/volumes/kubernetes.io~projected/kube-api-access-prpcz DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6fd964a7cdff5d40e64edc030cace79a4941c44ee1931f68ac96bd8628141cb/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ab5990d-5210-417d-80f9-d5d8bb0f7e03/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f4bc89ad-0df9-43d4-b142-9df5def7f3dd/volumes/kubernetes.io~projected/kube-api-access-qlnwn DeviceMajor:0 DeviceMinor:232 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/cf9f8413-6613-4db2-9dd1-20742c2af7ed/volumes/kubernetes.io~projected/kube-api-access-qtzxj DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4bc89ad-0df9-43d4-b142-9df5def7f3dd/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:229 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72778de9f271e583a7afed12182177aa39d73fb9aed049cb862ba0d20210617e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f9f28189-a808-48a4-b2da-d1950b84c7d0/volumes/kubernetes.io~projected/kube-api-access-mvswn DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8108f21a9c6fda01f5de299b4d89ec4742abc7f12f2850d8985c560bdb7b7665/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3ab5990d-5210-417d-80f9-d5d8bb0f7e03/volumes/kubernetes.io~projected/kube-api-access-dhvf9 DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9f28189-a808-48a4-b2da-d1950b84c7d0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caec9813de1596f33731e527d0e07328d7f9818c6240c19d6731d636a7b0f14b/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/723454fad717352ff4bf75bd39271d9f6143bf7379eb93c9e65534d7999cd976/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b0d94b94-63a5-404a-b169-e120292fcbe2/volumes/kubernetes.io~projected/kube-api-access-rckcg DeviceMajor:0 DeviceMinor:228 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba4c7dc1-2eea-456e-bc9a-b6d02814657f/volumes/kubernetes.io~projected/kube-api-access-zm6g2 DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ba7aa05fc2b28f32027396cda92af8499002e51a3e9b85eb294808a56270fb8/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d45df744d7ce685c3686ce232a54f70ef8a7a0f294edb4d561ad8baaedc13944/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fae2b865f46285a4c89708abf1924079adeb19e40f28370184b5a97935427b85/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8b2a6fcd-a217-4a03-994a-9261ed191c64/volumes/kubernetes.io~projected/kube-api-access-nnrpn DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f4950b203312825d4741862e4ed5d84fbac74b94028dab9c710e68b8733ddad/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81a0b663aae2a9e50f52cef108bf7d7f29c256ce7a85f067f346918dcd326d0b/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8b2a6fcd-a217-4a03-994a-9261ed191c64/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89e41adf-23e2-4eaa-a335-59002579bdce/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f31d9405-aac0-42bd-8c9a-977b6328bc65/volumes/kubernetes.io~projected/kube-api-access-q5l9h DeviceMajor:0 DeviceMinor:223 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8dd570f958fea83f1a71365ca5d691a1c681a593e138568b734ef95b893a24d/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff9ec61283d188f20a5694524f2017b347e14c6fab419a6cc14eb7ca36ba6a85/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8ed56df7-b752-4f2d-b941-0b7df5aa6df9/volumes/kubernetes.io~projected/kube-api-access-ks6h6 DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c/volumes/kubernetes.io~projected/kube-api-access-7vwsq DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055/volumes/kubernetes.io~projected/kube-api-access-s6jcz DeviceMajor:0 DeviceMinor:222 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fb1aebcff1ab1345a58c682d1bdd3ea96b0ddb540f8a968e6ecdc67e9e21fa7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:723454fad717352 MacAddress:5e:a2:a5:d5:32:d7 Speed:10000 Mtu:8901} {Name:81a0b663aae2a9e MacAddress:f6:19:1e:9a:98:2d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c9:dd:1d:c7 Speed:0 Mtu:9001} {Name:br-int MacAddress:da:a8:6b:37:a2:f5 Speed:0 Mtu:8901} {Name:d8dd570f958fea8 MacAddress:8e:ea:1a:f9:99:24 Speed:10000 Mtu:8901} {Name:ea9b3eba1954680 MacAddress:56:f6:55:ab:b9:83 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c9:dd:1d:c7 Speed:0 Mtu:9001} {Name:fae2b865f46285a MacAddress:d6:34:92:4d:bf:0c Speed:10000 Mtu:8901} {Name:ff9ec61283d188f MacAddress:2a:6d:5f:d6:73:6b Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:96:2d:31:a3:a8:89 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:fe:8d:6a:31:29:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 13:04:28.809679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.809500 4541 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 13:04:28.809679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.809570 4541 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 13:04:28.809905 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.809880 4541 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 13:04:28.810042 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.809906 4541 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-174.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 13:04:28.810116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810044 4541 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 13:04:28.810116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810053 4541 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 13:04:28.810116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810067 4541 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:04:28.810116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810082 4541 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:04:28.810233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810173 4541 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 13:04:28.810344 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810334 4541 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 13:04:28.810456 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810447 4541 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 13:04:28.810492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810484 4541 kubelet.go:493] "Attempting to sync node with API server" Nov 03 13:04:28.810520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810494 4541 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 13:04:28.810520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810505 4541 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 13:04:28.810520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810514 4541 kubelet.go:405] "Adding apiserver pod source" Nov 03 13:04:28.810687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.810522 4541 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 13:04:28.811185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.811172 4541 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 13:04:28.811185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.811185 4541 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:04:28.811711 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.811689 4541 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 13:04:28.825344 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.825310 4541 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 13:04:28.825344 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.825336 4541 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:04:28.825568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.825552 4541 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 13:04:28.826029 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826014 4541 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 13:04:28.826074 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826051 4541 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 13:04:28.826195 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826184 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 13:04:28.826233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826200 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 13:04:28.826233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826209 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 13:04:28.826233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826220 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 13:04:28.826233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826229 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 13:04:28.826233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826235 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 13:04:28.826382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826241 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 13:04:28.826382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826247 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 13:04:28.826382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826254 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 13:04:28.826382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826259 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 13:04:28.826382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826268 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 13:04:28.826382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826278 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 13:04:28.826382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826305 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 13:04:28.826382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.826315 4541 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 13:04:28.829813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.829801 4541 server.go:1263] "Started kubelet" Nov 03 13:04:28.830052 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.829893 4541 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 13:04:28.830052 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.829890 4541 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 13:04:28.830052 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.829992 4541 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 13:04:28.830269 ip-10-0-132-174 systemd[1]: Started Kubernetes Kubelet. Nov 03 13:04:28.831367 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.830322 4541 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 13:04:28.831367 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.830982 4541 server.go:317] "Adding debug handlers to kubelet server" Nov 03 13:04:28.832464 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.832449 4541 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 13:04:28.832547 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.832470 4541 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 13:04:28.832637 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.832576 4541 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 12:58:49 +0000 UTC" deadline="2028-06-29 00:11:18.22319301 +0000 UTC" Nov 03 13:04:28.832637 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.832611 4541 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14483h6m49.39058511s" Nov 03 13:04:28.832637 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.832613 4541 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 13:04:28.832637 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.832626 4541 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 13:04:28.832637 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.832628 4541 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 13:04:28.834455 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834415 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" volumeName="kubernetes.io/projected/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-kube-api-access-zm6g2" seLinuxMountContext="" Nov 03 13:04:28.834455 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834459 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf9f8413-6613-4db2-9dd1-20742c2af7ed" volumeName="kubernetes.io/projected/cf9f8413-6613-4db2-9dd1-20742c2af7ed-kube-api-access-qtzxj" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834473 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89e41adf-23e2-4eaa-a335-59002579bdce" volumeName="kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834486 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ed56df7-b752-4f2d-b941-0b7df5aa6df9" volumeName="kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-daemon-config" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834498 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" volumeName="kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-binary-copy" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834507 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e75c8fbb-02a6-4728-82cb-bf7cbe3e3055" volumeName="kubernetes.io/empty-dir/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-data-volume" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834514 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bb630b3-a283-4fa8-9887-1202daceae1a" volumeName="kubernetes.io/configmap/1bb630b3-a283-4fa8-9887-1202daceae1a-serviceca" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834522 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b2a6fcd-a217-4a03-994a-9261ed191c64" volumeName="kubernetes.io/projected/8b2a6fcd-a217-4a03-994a-9261ed191c64-kube-api-access-nnrpn" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834531 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ed56df7-b752-4f2d-b941-0b7df5aa6df9" volumeName="kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cni-binary-copy" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834539 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ed56df7-b752-4f2d-b941-0b7df5aa6df9" volumeName="kubernetes.io/projected/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-kube-api-access-ks6h6" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834547 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" volumeName="kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834554 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31d9405-aac0-42bd-8c9a-977b6328bc65" volumeName="kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834563 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31d9405-aac0-42bd-8c9a-977b6328bc65" volumeName="kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path" seLinuxMountContext="" Nov 03 13:04:28.834571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834571 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9f28189-a808-48a4-b2da-d1950b84c7d0" volumeName="kubernetes.io/projected/f9f28189-a808-48a4-b2da-d1950b84c7d0-kube-api-access-mvswn" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834584 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b2a6fcd-a217-4a03-994a-9261ed191c64" volumeName="kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-tmp" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834591 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8" volumeName="kubernetes.io/secret/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-agent-certs" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834600 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7dc614f-3d12-4003-b572-8dbcbfcf7b6b" volumeName="kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834607 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" volumeName="kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834614 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31d9405-aac0-42bd-8c9a-977b6328bc65" volumeName="kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834621 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9f28189-a808-48a4-b2da-d1950b84c7d0" volumeName="kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-env-overrides" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834629 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9f28189-a808-48a4-b2da-d1950b84c7d0" volumeName="kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-config" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834636 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ab5990d-5210-417d-80f9-d5d8bb0f7e03" volumeName="kubernetes.io/secret/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-cert" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834644 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48d92b6c-4577-458c-b6a7-7b11300ca6b1" volumeName="kubernetes.io/projected/48d92b6c-4577-458c-b6a7-7b11300ca6b1-kube-api-access-hwzw9" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834655 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b2a6fcd-a217-4a03-994a-9261ed191c64" volumeName="kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-tuned" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834666 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e75c8fbb-02a6-4728-82cb-bf7cbe3e3055" volumeName="kubernetes.io/projected/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-api-access-s6jcz" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834694 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f31d9405-aac0-42bd-8c9a-977b6328bc65" volumeName="kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834706 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4bc89ad-0df9-43d4-b142-9df5def7f3dd" volumeName="kubernetes.io/secret/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-key" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834717 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9f28189-a808-48a4-b2da-d1950b84c7d0" volumeName="kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-script-lib" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834754 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ab5990d-5210-417d-80f9-d5d8bb0f7e03" volumeName="kubernetes.io/projected/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-kube-api-access-dhvf9" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834766 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="48d92b6c-4577-458c-b6a7-7b11300ca6b1" volumeName="kubernetes.io/empty-dir/48d92b6c-4577-458c-b6a7-7b11300ca6b1-tmp-dir" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834792 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c" volumeName="kubernetes.io/configmap/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-iptables-alerter-script" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834820 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8" volumeName="kubernetes.io/configmap/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-konnectivity-ca" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834833 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9f28189-a808-48a4-b2da-d1950b84c7d0" volumeName="kubernetes.io/secret/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834845 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c6e919c-8135-4ac2-b2fd-aafb531180be" volumeName="kubernetes.io/projected/0c6e919c-8135-4ac2-b2fd-aafb531180be-kube-api-access-7824f" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834858 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0d94b94-63a5-404a-b169-e120292fcbe2" volumeName="kubernetes.io/projected/b0d94b94-63a5-404a-b169-e120292fcbe2-kube-api-access-rckcg" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834869 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d86486e7-2eb7-4200-a35b-8025870fc018" volumeName="kubernetes.io/projected/d86486e7-2eb7-4200-a35b-8025870fc018-kube-api-access-nnd4j" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834881 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4bc89ad-0df9-43d4-b142-9df5def7f3dd" volumeName="kubernetes.io/configmap/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-cabundle" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834893 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4bc89ad-0df9-43d4-b142-9df5def7f3dd" volumeName="kubernetes.io/projected/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-kube-api-access-qlnwn" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834904 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c6e919c-8135-4ac2-b2fd-aafb531180be" volumeName="kubernetes.io/empty-dir/0c6e919c-8135-4ac2-b2fd-aafb531180be-tmp-dir" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834915 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c" volumeName="kubernetes.io/projected/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-kube-api-access-7vwsq" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834928 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0c6e919c-8135-4ac2-b2fd-aafb531180be" volumeName="kubernetes.io/configmap/0c6e919c-8135-4ac2-b2fd-aafb531180be-config-volume" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834940 4541 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bb630b3-a283-4fa8-9887-1202daceae1a" volumeName="kubernetes.io/projected/1bb630b3-a283-4fa8-9887-1202daceae1a-kube-api-access-prpcz" seLinuxMountContext="" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834951 4541 reconstruct.go:97] "Volume reconstruction finished" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.834959 4541 reconciler.go:29] "Reconciler: start to sync state" Nov 03 13:04:28.836006 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:28.835040 4541 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:04:28.837044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.836668 4541 factory.go:55] Registering systemd factory Nov 03 13:04:28.837044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.836687 4541 factory.go:223] Registration of the systemd container factory successfully Nov 03 13:04:28.837801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.837222 4541 factory.go:153] Registering CRI-O factory Nov 03 13:04:28.837801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.837243 4541 factory.go:223] Registration of the crio container factory successfully Nov 03 13:04:28.837801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.837296 4541 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 13:04:28.837801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.837323 4541 factory.go:103] Registering Raw factory Nov 03 13:04:28.837801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.837339 4541 manager.go:1199] Started watching for new ooms in manager Nov 03 13:04:28.837801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.837699 4541 manager.go:319] Starting recovery of all containers Nov 03 13:04:28.847160 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.847135 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:04:28.847570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.847500 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:04:28.859112 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.859057 4541 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 13:04:28.860865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.860528 4541 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 13:04:28.860865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.860549 4541 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 13:04:28.860865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.860565 4541 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 13:04:28.860865 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:28.860597 4541 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 13:04:28.872371 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.872341 4541 generic.go:356] "Generic (PLEG): container finished" podID="747e2de570be6f488466605f00369eb2" containerID="f439e359a8800a2cf56f86288a9a5c4c3e95ff0c86530dfd5ecf56c318491213" exitCode=0 Nov 03 13:04:28.879126 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.879100 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:04:28.879530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.879502 4541 generic.go:356] "Generic (PLEG): container finished" podID="f9f28189-a808-48a4-b2da-d1950b84c7d0" containerID="bda8a73558cd0b85ca66dc4da57916070357bf07ab0a7fed38175cfe182ea03d" exitCode=1 Nov 03 13:04:28.902520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.902467 4541 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="0caef70fcfd9b913a4ef099b4bdc51c451280bb77ad10eb4f902706ecf9df993" exitCode=0 Nov 03 13:04:28.902520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.902505 4541 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="c8eb5381b01ed50139ec168d70e142d350ea16d55977c1227b41dc4a3951a265" exitCode=0 Nov 03 13:04:28.902520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.902516 4541 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="dd672ea77edd3418dd81027195e6a7b98e1aa245fe474e986553476620d03a8a" exitCode=0 Nov 03 13:04:28.902759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.902530 4541 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="c000d2841716b42e04539f13ee736d74c46fb1c76465cae073e066c25329ef26" exitCode=0 Nov 03 13:04:28.902759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.902539 4541 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="9ff868439c3c389c3405bf274ad3e45028fd1d9066147122b86de845227df8db" exitCode=0 Nov 03 13:04:28.902759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.902549 4541 generic.go:356] "Generic (PLEG): container finished" podID="ba4c7dc1-2eea-456e-bc9a-b6d02814657f" containerID="7f80d8a8055a2769099f9fa76d8333177d3dde24a2f25c574cee54d88bfc5e09" exitCode=0 Nov 03 13:04:28.910624 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.910604 4541 manager.go:324] Recovery completed Nov 03 13:04:28.925568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.925549 4541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:28.926876 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.926857 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:28.926963 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.926893 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:28.926963 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.926904 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:28.928353 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928338 4541 cpu_manager.go:226] "Starting" policy="none" Nov 03 13:04:28.928353 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928352 4541 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 13:04:28.928415 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928355 4541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:28.928415 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928368 4541 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 13:04:28.928501 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928486 4541 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 13:04:28.928574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928503 4541 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 13:04:28.928574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928524 4541 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 13:04:28.928574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928529 4541 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 13:04:28.928574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928535 4541 policy_none.go:50] "Start" Nov 03 13:04:28.928574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928543 4541 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 13:04:28.928574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928552 4541 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 13:04:28.928726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928686 4541 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 13:04:28.928726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928694 4541 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 13:04:28.928726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.928700 4541 policy_none.go:44] "Start" Nov 03 13:04:28.929293 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.929278 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:28.929334 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.929309 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:28.929334 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.929323 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:28.932743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.932711 4541 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 13:04:28.935901 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:28.935884 4541 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:04:28.936029 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.936019 4541 manager.go:341] "Starting Device Plugin manager" Nov 03 13:04:28.936072 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:28.936040 4541 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 13:04:28.936072 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.936048 4541 server.go:87] "Starting device plugin registration server" Nov 03 13:04:28.936289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.936278 4541 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 13:04:28.936347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.936290 4541 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 13:04:28.936403 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.936365 4541 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 13:04:28.936491 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.936457 4541 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 13:04:28.936556 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.936500 4541 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 13:04:28.936556 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.936507 4541 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 13:04:28.936646 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.936558 4541 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:04:28.936488703Z","UUID":"429101c7-8448-4564-87e5-0c3eae234032","Handler":null,"Name":"","Endpoint":""} Nov 03 13:04:28.937925 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:28.937898 4541 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 13:04:28.938001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.937932 4541 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 13:04:28.938001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.937962 4541 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 13:04:28.938114 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:28.937936 4541 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" Nov 03 13:04:28.961195 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.961154 4541 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal"] Nov 03 13:04:28.961318 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.961275 4541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:28.962523 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.962503 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:28.962617 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.962539 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:28.962617 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.962553 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:28.962905 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.962891 4541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:28.963033 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.963016 4541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:28.963642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.963626 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:28.963642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.963636 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:28.963724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.963658 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:28.963724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.963658 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:28.963724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.963673 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:28.963724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.963683 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:28.964165 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:28.964142 4541 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:28.964669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.964605 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"3ac4092c55873e071f156a04a17c98a5","Type":"ContainerStarted","Data":"5e0af216ce38053eef0872c0b06eb860018a48d37240953dc242023fd46d625c"} Nov 03 13:04:28.964743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.964687 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"3ac4092c55873e071f156a04a17c98a5","Type":"ContainerStarted","Data":"72778de9f271e583a7afed12182177aa39d73fb9aed049cb862ba0d20210617e"} Nov 03 13:04:28.964743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.964709 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerStarted","Data":"a309afd6a252b66a1129aa5ccf3931a9ec2f2cc20a4bf303ed8de3b352c062ff"} Nov 03 13:04:28.964743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.964737 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerDied","Data":"f439e359a8800a2cf56f86288a9a5c4c3e95ff0c86530dfd5ecf56c318491213"} Nov 03 13:04:28.964902 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.964758 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" event={"ID":"747e2de570be6f488466605f00369eb2","Type":"ContainerStarted","Data":"1fb1aebcff1ab1345a58c682d1bdd3ea96b0ddb540f8a968e6ecdc67e9e21fa7"} Nov 03 13:04:28.964991 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.964976 4541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:28.966137 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.966116 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:28.966212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.966158 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:28.966212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:28.966176 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:28.966393 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:28.966372 4541 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-174.us-west-2.compute.internal\" not found" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.011273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.011237 4541 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:29.011569 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.011550 4541 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:29.011657 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.011571 4541 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 13:04:29.011657 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.011580 4541 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:29.035845 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.035814 4541 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.036902 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.036882 4541 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:29.038210 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.038194 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:29.038294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.038223 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:29.038294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.038233 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:29.038294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.038275 4541 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.060949 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.060919 4541 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:29.104387 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.104353 4541 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:04:29.104558 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:29.104409 4541 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.104558 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.104424 4541 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.112400 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.112339 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.112400 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.112369 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.112400 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.112390 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.153444 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.153411 4541 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.153574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.153490 4541 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.154080 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.154062 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeNotReady" Nov 03 13:04:29.154152 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.154094 4541 setters.go:546] "Node became not ready" node="ip-10-0-132-174.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T13:04:29Z","lastTransitionTime":"2026-11-03T13:04:29Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 13:04:29.161825 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.161807 4541 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:04:29.161993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.161832 4541 nodeinfomanager.go:471] existing CSINode "ip-10-0-132-174.us-west-2.compute.internal" is owned by different node (oldNodeID="2bd08222-4005-421d-a345-ead18adeab43", newNodeID=""), cleaning up... Nov 03 13:04:29.161993 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:29.161855 4541 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.212632 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.212599 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.212632 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.212633 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.212840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.212650 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.212840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.212694 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.212840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.212713 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/747e2de570be6f488466605f00369eb2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"747e2de570be6f488466605f00369eb2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.212840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.212737 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ac4092c55873e071f156a04a17c98a5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal\" (UID: \"3ac4092c55873e071f156a04a17c98a5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.811540 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.811469 4541 apiserver.go:52] "Watching apiserver" Nov 03 13:04:29.905240 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.905214 4541 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.910389 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.910374 4541 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 13:04:29.910859 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.910840 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b","openshift-ovn-kubernetes/ovnkube-node-5sjz9","openshift-network-diagnostics/network-check-target-hfbh8","openshift-insights/insights-runtime-extractor-pwsr8","openshift-cluster-node-tuning-operator/tuned-mjx8m","openshift-multus/multus-j99cl","openshift-network-operator/iptables-alerter-k8dl6","openshift-multus/multus-additional-cni-plugins-qm2ff","openshift-multus/network-metrics-daemon-r8f65","openshift-image-registry/node-ca-tgrzf","openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal","openshift-dns/node-resolver-5zlvj","openshift-service-ca/service-ca-5fffc97bdf-29qhq","kube-system/konnectivity-agent-pjnlk","openshift-dns/dns-default-b5n5d","openshift-ingress-canary/ingress-canary-pnc5z","openshift-insights/periodic-gathering-9xmlk-2g7w7","kube-system/global-pull-secret-syncer-ncrq2"] Nov 03 13:04:29.917629 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.917605 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:29.918804 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.918786 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:29.920107 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.920089 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:29.934099 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.934079 4541 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 13:04:29.961699 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961675 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 13:04:29.961699 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961673 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 13:04:29.961907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961713 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 13:04:29.961907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961804 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nbm\"" Nov 03 13:04:29.961907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961828 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 13:04:29.961907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961835 4541 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:04:29.961907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961856 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 13:04:29.961907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961867 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-sctm8\"" Nov 03 13:04:29.961907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961877 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 13:04:29.961907 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:29.961892 4541 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-174.us-west-2.compute.internal" Nov 03 13:04:29.962159 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961909 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 13:04:29.962159 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961913 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 13:04:29.962159 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961953 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 13:04:29.962159 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.961828 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lbs9x\"" Nov 03 13:04:29.962159 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:29.962031 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 13:04:30.016691 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.016662 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bb630b3-a283-4fa8-9887-1202daceae1a-serviceca\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:04:30.016691 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.016693 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-systemd-units\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.016899 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.016711 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnwn\" (UniqueName: \"kubernetes.io/projected/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-kube-api-access-qlnwn\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:30.016899 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.016728 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-system-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.016899 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.016745 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-node-log\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.017039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.016912 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.017039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.016955 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-script-lib\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.017039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.016984 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:30.017039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017014 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-dbus\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:30.017039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017037 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-tmp\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.017232 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017074 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:30.017232 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017127 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-tmp\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.017232 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017225 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-multus-certs\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.017371 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017254 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-modprobe-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.017371 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017274 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-conf\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.017371 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017293 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-tuned\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.017371 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017308 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.017371 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017339 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnd4j\" (UniqueName: \"kubernetes.io/projected/d86486e7-2eb7-4200-a35b-8025870fc018-kube-api-access-nnd4j\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.017600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017402 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-host-slash\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:04:30.017600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017423 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-tuned\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.017600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017412 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-script-lib\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.017600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017459 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-daemon-config\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.017600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017540 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-sys\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.017600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017559 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtzxj\" (UniqueName: \"kubernetes.io/projected/cf9f8413-6613-4db2-9dd1-20742c2af7ed-kube-api-access-qtzxj\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:30.017600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017589 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-cert\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017621 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-cabundle\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017658 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017701 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017729 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cni-binary-copy\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017756 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-kube-api-access-ks6h6\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017759 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017797 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-ovn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017828 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017898 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysconfig\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.017948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017940 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrpn\" (UniqueName: \"kubernetes.io/projected/8b2a6fcd-a217-4a03-994a-9261ed191c64-kube-api-access-nnrpn\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.018270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017965 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6e919c-8135-4ac2-b2fd-aafb531180be-tmp-dir\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:30.018270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.017984 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:30.018270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018142 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-multus\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.018270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018151 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6e919c-8135-4ac2-b2fd-aafb531180be-tmp-dir\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:30.018270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018197 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-slash\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.018270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018227 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-config\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.018270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018256 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-var-lib-kubelet\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.018546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018280 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-sys-fs\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.018546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018325 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vwsq\" (UniqueName: \"kubernetes.io/projected/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-kube-api-access-7vwsq\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:04:30.018546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018374 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cnibin\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.018546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018420 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovnkube-config\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.018546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018474 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb630b3-a283-4fa8-9887-1202daceae1a-host\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:04:30.018546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018500 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-konnectivity-ca\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:04:30.018546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018530 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-bin\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.018862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018597 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvswn\" (UniqueName: \"kubernetes.io/projected/f9f28189-a808-48a4-b2da-d1950b84c7d0-kube-api-access-mvswn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.018862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018699 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.018862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018739 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-kubernetes\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.018862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018764 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.019018 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018871 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-kubelet\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.019018 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018892 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-konnectivity-ca\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:04:30.019018 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018902 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-kubelet\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.019018 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.018957 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-log-socket\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.019209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019019 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm6g2\" (UniqueName: \"kubernetes.io/projected/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-kube-api-access-zm6g2\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.019209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019054 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5l9h\" (UniqueName: \"kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:30.019209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019082 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.019209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019109 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-bin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.019209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019136 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:30.019209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019166 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.019209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019200 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-netns\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019251 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-hostroot\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019281 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-conf-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019311 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prpcz\" (UniqueName: \"kubernetes.io/projected/1bb630b3-a283-4fa8-9887-1202daceae1a-kube-api-access-prpcz\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019333 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d92b6c-4577-458c-b6a7-7b11300ca6b1-tmp-dir\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019348 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-netd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019363 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019425 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-kubelet-config\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019442 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48d92b6c-4577-458c-b6a7-7b11300ca6b1-tmp-dir\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019458 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-agent-certs\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:04:30.019564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019535 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019577 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-systemd\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019604 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-registration-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019650 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7824f\" (UniqueName: \"kubernetes.io/projected/0c6e919c-8135-4ac2-b2fd-aafb531180be-kube-api-access-7824f\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019739 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvf9\" (UniqueName: \"kubernetes.io/projected/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-kube-api-access-dhvf9\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019791 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-data-volume\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019816 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-env-overrides\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019836 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9f28189-a808-48a4-b2da-d1950b84c7d0-ovn-node-metrics-cert\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019793 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8-agent-certs\") pod \"konnectivity-agent-pjnlk\" (UID: \"91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8\") " pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019851 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-lib-modules\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.019989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.019861 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-data-volume\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.020259 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020236 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9f28189-a808-48a4-b2da-d1950b84c7d0-env-overrides\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.020446 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020431 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-etc-selinux\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.020483 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020457 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.020516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020476 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.020516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020507 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6jcz\" (UniqueName: \"kubernetes.io/projected/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-api-access-s6jcz\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.020573 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020543 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-socket-dir-parent\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.020658 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020645 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-systemd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.020692 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020682 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-device-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.020725 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020712 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-crio-socket\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.020754 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020739 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rckcg\" (UniqueName: \"kubernetes.io/projected/b0d94b94-63a5-404a-b169-e120292fcbe2-kube-api-access-rckcg\") pod \"migrator-66df5d496b-2rzqk\" (UID: \"b0d94b94-63a5-404a-b169-e120292fcbe2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" Nov 03 13:04:30.020800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020759 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-iptables-alerter-script\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:04:30.020847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020830 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-system-cni-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.020881 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020858 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.020936 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020903 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-key\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:30.020992 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.020969 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:30.021024 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021000 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.021024 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021019 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-host\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.021118 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021040 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6e919c-8135-4ac2-b2fd-aafb531180be-config-volume\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:30.021118 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021071 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cnibin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.021215 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021129 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-etc-kubernetes\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.021215 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021162 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-netns\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.021215 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021190 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-etc-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.021306 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021218 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-run\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.021306 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021245 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-socket-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.021306 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021266 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-var-lib-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.021306 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021294 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48d92b6c-4577-458c-b6a7-7b11300ca6b1-hosts-file\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:04:30.021423 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021322 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzw9\" (UniqueName: \"kubernetes.io/projected/48d92b6c-4577-458c-b6a7-7b11300ca6b1-kube-api-access-hwzw9\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:04:30.021461 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.021446 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-k8s-cni-cncf-io\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.061719 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.061649 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.061932 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.061912 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 13:04:30.062002 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.061974 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 13:04:30.062053 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.061995 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 13:04:30.062099 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062068 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g4lr6\"" Nov 03 13:04:30.062206 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062158 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.062324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062239 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 13:04:30.062324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062292 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.062454 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062326 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m8wc5\"" Nov 03 13:04:30.062454 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062421 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-697z8\"" Nov 03 13:04:30.062639 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062615 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 13:04:30.062738 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062723 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5wxnl\"" Nov 03 13:04:30.062840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062811 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l9fl5\"" Nov 03 13:04:30.062840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062831 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 13:04:30.062840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062834 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.062990 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062906 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 13:04:30.062990 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.062938 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 13:04:30.063112 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063066 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9jvp2\"" Nov 03 13:04:30.063176 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063131 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 13:04:30.063176 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063164 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.063176 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063173 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 13:04:30.063346 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063181 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.063346 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063181 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.063346 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063190 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8p2l7\"" Nov 03 13:04:30.063346 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063341 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.063505 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063476 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.063553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063477 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 13:04:30.063553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063512 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.063553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063516 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 13:04:30.063553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063529 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.063553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063545 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.063553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063530 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063556 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063589 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063596 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063601 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063591 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063629 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063648 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnhgg\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063662 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063665 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063678 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pvm6f\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063684 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063693 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5ntlc\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063708 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8v28q\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063780 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.063877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.063835 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 13:04:30.067187 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.067167 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bb630b3-a283-4fa8-9887-1202daceae1a-serviceca\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:04:30.068270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.068254 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-cabundle\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:30.068366 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.068306 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-cert\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:30.068366 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.068330 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cni-binary-copy\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.068366 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.068337 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-daemon-config\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.068366 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.068307 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:30.068496 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.068377 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/89e41adf-23e2-4eaa-a335-59002579bdce-original-pull-secret\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:30.069428 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.069409 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.069428 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.069421 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.071618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.071597 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0c6e919c-8135-4ac2-b2fd-aafb531180be-config-volume\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:30.071673 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.071626 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-signing-key\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:30.071673 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.071639 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-iptables-alerter-script\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:04:30.071673 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.071624 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cni-binary-copy\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.108450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.108418 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-85kzz\"" Nov 03 13:04:30.108593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.108502 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s6pph\"" Nov 03 13:04:30.108593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.108531 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pvzn\"" Nov 03 13:04:30.108593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.108581 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 13:04:30.108687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.108588 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xbj7f\"" Nov 03 13:04:30.112802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.111472 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 13:04:30.112802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.111990 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtzxj\" (UniqueName: \"kubernetes.io/projected/cf9f8413-6613-4db2-9dd1-20742c2af7ed-kube-api-access-qtzxj\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:30.112802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.112015 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vwsq\" (UniqueName: \"kubernetes.io/projected/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-kube-api-access-7vwsq\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:04:30.112802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.112052 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5l9h\" (UniqueName: \"kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:30.112802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.112374 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnwn\" (UniqueName: \"kubernetes.io/projected/f4bc89ad-0df9-43d4-b142-9df5def7f3dd-kube-api-access-qlnwn\") pod \"service-ca-5fffc97bdf-29qhq\" (UID: \"f4bc89ad-0df9-43d4-b142-9df5def7f3dd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" Nov 03 13:04:30.112802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.112427 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6jcz\" (UniqueName: \"kubernetes.io/projected/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-api-access-s6jcz\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.112802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.112693 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prpcz\" (UniqueName: \"kubernetes.io/projected/1bb630b3-a283-4fa8-9887-1202daceae1a-kube-api-access-prpcz\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:04:30.112802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.112758 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrpn\" (UniqueName: \"kubernetes.io/projected/8b2a6fcd-a217-4a03-994a-9261ed191c64-kube-api-access-nnrpn\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.113297 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.112936 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm6g2\" (UniqueName: \"kubernetes.io/projected/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-kube-api-access-zm6g2\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.113297 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.113030 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnd4j\" (UniqueName: \"kubernetes.io/projected/d86486e7-2eb7-4200-a35b-8025870fc018-kube-api-access-nnd4j\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.113408 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.113352 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvswn\" (UniqueName: \"kubernetes.io/projected/f9f28189-a808-48a4-b2da-d1950b84c7d0-kube-api-access-mvswn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.113552 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.113472 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwzw9\" (UniqueName: \"kubernetes.io/projected/48d92b6c-4577-458c-b6a7-7b11300ca6b1-kube-api-access-hwzw9\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:04:30.113552 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.113488 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmhgn\" (UniqueName: \"kubernetes.io/projected/a7dc614f-3d12-4003-b572-8dbcbfcf7b6b-kube-api-access-nmhgn\") pod \"network-check-target-hfbh8\" (UID: \"a7dc614f-3d12-4003-b572-8dbcbfcf7b6b\") " pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:30.113689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.113591 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvf9\" (UniqueName: \"kubernetes.io/projected/3ab5990d-5210-417d-80f9-d5d8bb0f7e03-kube-api-access-dhvf9\") pod \"ingress-canary-pnc5z\" (UID: \"3ab5990d-5210-417d-80f9-d5d8bb0f7e03\") " pod="openshift-ingress-canary/ingress-canary-pnc5z" Nov 03 13:04:30.114857 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.113845 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7824f\" (UniqueName: \"kubernetes.io/projected/0c6e919c-8135-4ac2-b2fd-aafb531180be-kube-api-access-7824f\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:30.114857 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.113961 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rckcg\" (UniqueName: \"kubernetes.io/projected/b0d94b94-63a5-404a-b169-e120292fcbe2-kube-api-access-rckcg\") pod \"migrator-66df5d496b-2rzqk\" (UID: \"b0d94b94-63a5-404a-b169-e120292fcbe2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2rzqk" Nov 03 13:04:30.114857 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.113999 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-kube-api-access-ks6h6\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.121651 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.121629 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle\") pod \"periodic-gathering-9xmlk-2g7w7\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:04:30.121854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.121840 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-k8s-cni-cncf-io\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.121920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.121862 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-systemd-units\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.121920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.121898 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-system-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.121920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.121914 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-node-log\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122042 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.121940 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122042 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.121952 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-k8s-cni-cncf-io\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122042 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.121965 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-dbus\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:30.122042 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.121969 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-systemd-units\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122042 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122011 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-node-log\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122042 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122013 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-system-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122016 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122023 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-dbus\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:30.122205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122054 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:30.122205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122100 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-multus-certs\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122127 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-modprobe-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122205 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.122138 4541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:30.122205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122149 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-conf\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122150 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-multus-certs\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122203 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.122227 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:04:30.622208076 +0000 UTC m=+1.936179936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : secret "metrics-daemon-secret" not found Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122243 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122254 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-host-slash\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122264 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-modprobe-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122279 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-sys\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122287 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-conf\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122305 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c-host-slash\") pod \"iptables-alerter-k8dl6\" (UID: \"8b0e9fa9-e8f0-4e70-ae9b-3fae77f8c07c\") " pod="openshift-network-operator/iptables-alerter-k8dl6" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122307 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122335 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-ovn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122339 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-sys\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122357 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysconfig\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122375 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-multus\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122389 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-slash\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122402 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-var-lib-kubelet\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122402 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-ovn\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122416 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-sys-fs\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122417 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysconfig\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122434 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cnibin\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122448 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb630b3-a283-4fa8-9887-1202daceae1a-host\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122451 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122458 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-slash\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122471 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-bin\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122488 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-sys-fs\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122495 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-var-lib-kubelet\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122501 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bb630b3-a283-4fa8-9887-1202daceae1a-host\") pod \"node-ca-tgrzf\" (UID: \"1bb630b3-a283-4fa8-9887-1202daceae1a\") " pod="openshift-image-registry/node-ca-tgrzf" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122502 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-cnibin\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122496 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-kubernetes\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122514 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-multus\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122542 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122549 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-kubernetes\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122562 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-kubelet\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122569 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-bin\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122577 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-kubelet\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122597 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-log-socket\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122615 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122617 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-kubelet\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122622 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-log-socket\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122631 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-bin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122640 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-kubelet\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122657 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-var-lib-cni-bin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122680 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122696 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-netns\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122725 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-hostroot\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122740 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-conf-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.122758 4541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122763 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.122811 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls podName:0c6e919c-8135-4ac2-b2fd-aafb531180be nodeName:}" failed. No retries permitted until 2026-11-03 13:04:30.622801078 +0000 UTC m=+1.936772915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls") pod "dns-default-b5n5d" (UID: "0c6e919c-8135-4ac2-b2fd-aafb531180be") : secret "dns-default-metrics-tls" not found Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122811 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-cni-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122826 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-host-run-netns\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122855 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-conf-dir\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122874 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-hostroot\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122894 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-netd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.122892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122913 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-cni-netd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122915 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122934 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-kubelet-config\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122956 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-systemd\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122974 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-registration-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122990 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-lib-modules\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.122998 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-sysctl-d\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123005 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-etc-selinux\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123024 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-etc-systemd\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123045 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123061 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123076 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-socket-dir-parent\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123114 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-registration-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123124 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-systemd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123151 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-device-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123159 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-systemd\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123161 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-lib-modules\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123161 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-multus-socket-dir-parent\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123176 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-crio-socket\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123154 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-etc-selinux\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123199 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-device-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123213 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-system-cni-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123215 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-crio-socket\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123186 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/89e41adf-23e2-4eaa-a335-59002579bdce-kubelet-config\") pod \"global-pull-secret-syncer-ncrq2\" (UID: \"89e41adf-23e2-4eaa-a335-59002579bdce\") " pod="kube-system/global-pull-secret-syncer-ncrq2" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123238 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123244 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ba4c7dc1-2eea-456e-bc9a-b6d02814657f-system-cni-dir\") pod \"multus-additional-cni-plugins-qm2ff\" (UID: \"ba4c7dc1-2eea-456e-bc9a-b6d02814657f\") " pod="openshift-multus/multus-additional-cni-plugins-qm2ff" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123252 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-host\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123264 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-run-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123277 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cnibin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.123232 4541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123296 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-host\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123301 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-etc-kubernetes\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123325 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-netns\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.123337 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:30.623323226 +0000 UTC m=+1.937295071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123342 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-cnibin\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123351 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-host-run-netns\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123356 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed56df7-b752-4f2d-b941-0b7df5aa6df9-etc-kubernetes\") pod \"multus-j99cl\" (UID: \"8ed56df7-b752-4f2d-b941-0b7df5aa6df9\") " pod="openshift-multus/multus-j99cl" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123371 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-etc-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123390 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-run\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123404 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-socket-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123430 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-var-lib-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123447 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48d92b6c-4577-458c-b6a7-7b11300ca6b1-hosts-file\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123455 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b2a6fcd-a217-4a03-994a-9261ed191c64-run\") pod \"tuned-mjx8m\" (UID: \"8b2a6fcd-a217-4a03-994a-9261ed191c64\") " pod="openshift-cluster-node-tuning-operator/tuned-mjx8m" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123477 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d86486e7-2eb7-4200-a35b-8025870fc018-socket-dir\") pod \"aws-ebs-csi-driver-node-9k77b\" (UID: \"d86486e7-2eb7-4200-a35b-8025870fc018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9k77b" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123490 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-var-lib-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123494 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9f28189-a808-48a4-b2da-d1950b84c7d0-etc-openvswitch\") pod \"ovnkube-node-5sjz9\" (UID: \"f9f28189-a808-48a4-b2da-d1950b84c7d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123521 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48d92b6c-4577-458c-b6a7-7b11300ca6b1-hosts-file\") pod \"node-resolver-5zlvj\" (UID: \"48d92b6c-4577-458c-b6a7-7b11300ca6b1\") " pod="openshift-dns/node-resolver-5zlvj" Nov 03 13:04:30.124192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.123650 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.222284 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.222264 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:04:30.318355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.318285 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:30.628193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.628106 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:30.628193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.628164 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:30.628425 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:30.628198 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:30.628425 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.628256 4541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:30.628425 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.628328 4541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:30.628425 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.628361 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls podName:0c6e919c-8135-4ac2-b2fd-aafb531180be nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.628347467 +0000 UTC m=+2.942319322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls") pod "dns-default-b5n5d" (UID: "0c6e919c-8135-4ac2-b2fd-aafb531180be") : secret "dns-default-metrics-tls" not found Nov 03 13:04:30.628425 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.628376 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.628369781 +0000 UTC m=+2.942341619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:30.628425 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.628375 4541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:30.628629 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:30.628449 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:04:31.628433565 +0000 UTC m=+2.942405420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : secret "metrics-daemon-secret" not found Nov 03 13:04:31.298887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:31.298868 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5zlvj_48d92b6c-4577-458c-b6a7-7b11300ca6b1/dns-node-resolver/0.log" Nov 03 13:04:31.637937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:31.637844 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:31.637937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:31.637903 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:31.637937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:31.637930 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:31.638170 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:31.638005 4541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:31.638170 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:31.638041 4541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:31.638170 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:31.638011 4541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:31.638170 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:31.638081 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:04:33.638063125 +0000 UTC m=+4.952034988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : secret "metrics-daemon-secret" not found Nov 03 13:04:31.638170 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:31.638117 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls podName:0c6e919c-8135-4ac2-b2fd-aafb531180be nodeName:}" failed. No retries permitted until 2026-11-03 13:04:33.638104345 +0000 UTC m=+4.952076184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls") pod "dns-default-b5n5d" (UID: "0c6e919c-8135-4ac2-b2fd-aafb531180be") : secret "dns-default-metrics-tls" not found Nov 03 13:04:31.638170 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:31.638133 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:33.638127046 +0000 UTC m=+4.952098883 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:31.911993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:31.911961 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-29qhq" event={"ID":"f4bc89ad-0df9-43d4-b142-9df5def7f3dd","Type":"ContainerStarted","Data":"646c74ed37587678cf870dba4d850b1386bb26e518e721403d2215a96843f277"} Nov 03 13:04:32.498813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:32.498766 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tgrzf_1bb630b3-a283-4fa8-9887-1202daceae1a/node-ca/0.log" Nov 03 13:04:33.299167 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:33.299136 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pnc5z_3ab5990d-5210-417d-80f9-d5d8bb0f7e03/serve-healthcheck-canary/0.log" Nov 03 13:04:33.499709 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:33.499678 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-2rzqk_b0d94b94-63a5-404a-b169-e120292fcbe2/migrator/0.log" Nov 03 13:04:33.654187 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:33.654157 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:33.654347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:33.654215 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:33.654347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:33.654250 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:33.654347 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:33.654321 4541 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 13:04:33.654457 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:33.654353 4541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:33.654457 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:33.654364 4541 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 13:04:33.654457 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:33.654397 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls podName:0c6e919c-8135-4ac2-b2fd-aafb531180be nodeName:}" failed. No retries permitted until 2026-11-03 13:04:37.654382776 +0000 UTC m=+8.968354615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls") pod "dns-default-b5n5d" (UID: "0c6e919c-8135-4ac2-b2fd-aafb531180be") : secret "dns-default-metrics-tls" not found Nov 03 13:04:33.654457 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:33.654421 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls podName:e75c8fbb-02a6-4728-82cb-bf7cbe3e3055 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:37.654404429 +0000 UTC m=+8.968376269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pwsr8" (UID: "e75c8fbb-02a6-4728-82cb-bf7cbe3e3055") : secret "insights-runtime-extractor-tls" not found Nov 03 13:04:33.654457 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:33.654436 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:04:37.654430251 +0000 UTC m=+8.968402089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : secret "metrics-daemon-secret" not found Nov 03 13:04:34.924043 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:34.924017 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:34.924442 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:34.924196 4541 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:04:37.485822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:37.485789 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5sjz9" Nov 03 13:04:37.682832 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:37.682794 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:37.683013 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:37.682848 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:37.683013 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:37.682895 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:37.683013 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:37.682944 4541 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 13:04:37.683134 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:37.683030 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs podName:cf9f8413-6613-4db2-9dd1-20742c2af7ed nodeName:}" failed. No retries permitted until 2026-11-03 13:04:45.683011871 +0000 UTC m=+16.996983720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs") pod "network-metrics-daemon-r8f65" (UID: "cf9f8413-6613-4db2-9dd1-20742c2af7ed") : secret "metrics-daemon-secret" not found Nov 03 13:04:37.683243 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:37.683225 4541 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 13:04:37.685251 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:37.685222 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e75c8fbb-02a6-4728-82cb-bf7cbe3e3055-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pwsr8\" (UID: \"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055\") " pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:37.685408 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:37.685320 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0c6e919c-8135-4ac2-b2fd-aafb531180be-metrics-tls\") pod \"dns-default-b5n5d\" (UID: \"0c6e919c-8135-4ac2-b2fd-aafb531180be\") " pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:37.719356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:37.719319 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:37.720412 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:37.720394 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pwsr8" Nov 03 13:04:38.089052 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:38.088973 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b5n5d"] Nov 03 13:04:38.092242 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:38.092215 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6e919c_8135_4ac2_b2fd_aafb531180be.slice/crio-69ab4b6c9aee2701345b2d643195487d5bab24712df1dac25050e40f13947828 WatchSource:0}: Error finding container 69ab4b6c9aee2701345b2d643195487d5bab24712df1dac25050e40f13947828: Status 404 returned error can't find the container with id 69ab4b6c9aee2701345b2d643195487d5bab24712df1dac25050e40f13947828 Nov 03 13:04:38.134179 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:38.134126 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pwsr8"] Nov 03 13:04:38.137923 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:38.137896 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75c8fbb_02a6_4728_82cb_bf7cbe3e3055.slice/crio-371616232c903df19602b57b30fbe829b6cfaee1b87a6966eb7509d65338b0df WatchSource:0}: Error finding container 371616232c903df19602b57b30fbe829b6cfaee1b87a6966eb7509d65338b0df: Status 404 returned error can't find the container with id 371616232c903df19602b57b30fbe829b6cfaee1b87a6966eb7509d65338b0df Nov 03 13:04:38.931615 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:38.931576 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pwsr8" event={"ID":"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055","Type":"ContainerStarted","Data":"90fe6b9b4ced01ffc87321109719696865e939b28dc13abde7e267d7553f6106"} Nov 03 13:04:38.932039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:38.931639 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pwsr8" event={"ID":"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055","Type":"ContainerStarted","Data":"371616232c903df19602b57b30fbe829b6cfaee1b87a6966eb7509d65338b0df"} Nov 03 13:04:38.932689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:38.932660 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b5n5d" event={"ID":"0c6e919c-8135-4ac2-b2fd-aafb531180be","Type":"ContainerStarted","Data":"69ab4b6c9aee2701345b2d643195487d5bab24712df1dac25050e40f13947828"} Nov 03 13:04:39.349220 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:39.349146 4541 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-174.us-west-2.compute.internal" event="NodeReady" Nov 03 13:04:39.726364 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:39.726319 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:04:39.938287 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:39.938243 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pwsr8" event={"ID":"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055","Type":"ContainerStarted","Data":"ef19b1b7a8f874f6fc4285654d11f3f879bf592f71f5361ac669e3050e007770"} Nov 03 13:04:40.946726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:40.946687 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b5n5d" event={"ID":"0c6e919c-8135-4ac2-b2fd-aafb531180be","Type":"ContainerStarted","Data":"339e036d41582e9c45c3945a5bb611fae48e1c9f01b3c276659df2e9e9c2ec6c"} Nov 03 13:04:40.947154 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:40.946755 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b5n5d" event={"ID":"0c6e919c-8135-4ac2-b2fd-aafb531180be","Type":"ContainerStarted","Data":"b71abbbe1c656ed096c9d1415d40873fdbd2a8e3b4cadfe7d36914e1c4179ce4"} Nov 03 13:04:40.947154 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:40.946818 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:41.674832 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:41.674803 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pjnlk" Nov 03 13:04:41.952321 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:41.952226 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pwsr8" event={"ID":"e75c8fbb-02a6-4728-82cb-bf7cbe3e3055","Type":"ContainerStarted","Data":"fe92d0981ae2122bdb57f2d594065433d59e0ddb5a2fc5a0bf96b49a1e07b643"} Nov 03 13:04:42.536659 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.536623 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88"] Nov 03 13:04:42.538493 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.538476 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" Nov 03 13:04:42.591297 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.591257 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 13:04:42.591492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.591471 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cljzb\"" Nov 03 13:04:42.594931 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.594905 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88"] Nov 03 13:04:42.618399 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.618368 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bea2f07b-470d-4c61-8740-989b36f04c36-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2fp88\" (UID: \"bea2f07b-470d-4c61-8740-989b36f04c36\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" Nov 03 13:04:42.633500 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.633476 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d59d8bb6c-87rbn"] Nov 03 13:04:42.635331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.635310 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.719527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.719492 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49d6484b-12c8-4554-85cd-fb0d0b2814f2-ca-trust-extracted\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.719527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.719524 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49d6484b-12c8-4554-85cd-fb0d0b2814f2-trusted-ca\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.719741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.719546 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49d6484b-12c8-4554-85cd-fb0d0b2814f2-installation-pull-secrets\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.719741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.719686 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d6484b-12c8-4554-85cd-fb0d0b2814f2-registry-tls\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.719741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.719723 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49d6484b-12c8-4554-85cd-fb0d0b2814f2-image-registry-private-configuration\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.719898 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.719745 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49d6484b-12c8-4554-85cd-fb0d0b2814f2-registry-certificates\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.719898 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.719762 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49d6484b-12c8-4554-85cd-fb0d0b2814f2-bound-sa-token\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.719898 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.719814 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bea2f07b-470d-4c61-8740-989b36f04c36-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2fp88\" (UID: \"bea2f07b-470d-4c61-8740-989b36f04c36\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" Nov 03 13:04:42.719986 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:42.719903 4541 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Nov 03 13:04:42.719986 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.719947 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmsw\" (UniqueName: \"kubernetes.io/projected/49d6484b-12c8-4554-85cd-fb0d0b2814f2-kube-api-access-6tmsw\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.719986 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:42.719962 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea2f07b-470d-4c61-8740-989b36f04c36-tls-certificates podName:bea2f07b-470d-4c61-8740-989b36f04c36 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:43.219948635 +0000 UTC m=+14.533920501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/bea2f07b-470d-4c61-8740-989b36f04c36-tls-certificates") pod "prometheus-operator-admission-webhook-6489d8bf89-2fp88" (UID: "bea2f07b-470d-4c61-8740-989b36f04c36") : secret "prometheus-operator-admission-webhook-tls" not found Nov 03 13:04:42.734529 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.734505 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 13:04:42.735853 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.735835 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 13:04:42.746619 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.746595 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 13:04:42.746822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.746648 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 13:04:42.746822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.746733 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5prrc\"" Nov 03 13:04:42.747730 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.747712 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d59d8bb6c-87rbn"] Nov 03 13:04:42.821064 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.820984 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmsw\" (UniqueName: \"kubernetes.io/projected/49d6484b-12c8-4554-85cd-fb0d0b2814f2-kube-api-access-6tmsw\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.821064 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.821032 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49d6484b-12c8-4554-85cd-fb0d0b2814f2-ca-trust-extracted\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.821064 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.821049 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49d6484b-12c8-4554-85cd-fb0d0b2814f2-trusted-ca\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.821361 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.821076 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49d6484b-12c8-4554-85cd-fb0d0b2814f2-installation-pull-secrets\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.821361 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.821111 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d6484b-12c8-4554-85cd-fb0d0b2814f2-registry-tls\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.821361 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.821133 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49d6484b-12c8-4554-85cd-fb0d0b2814f2-image-registry-private-configuration\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.821640 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.821598 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49d6484b-12c8-4554-85cd-fb0d0b2814f2-ca-trust-extracted\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.821640 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.821626 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49d6484b-12c8-4554-85cd-fb0d0b2814f2-registry-certificates\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.821862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.821662 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49d6484b-12c8-4554-85cd-fb0d0b2814f2-bound-sa-token\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.822118 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.822097 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49d6484b-12c8-4554-85cd-fb0d0b2814f2-registry-certificates\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.822451 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.822431 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49d6484b-12c8-4554-85cd-fb0d0b2814f2-trusted-ca\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.823748 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.823721 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49d6484b-12c8-4554-85cd-fb0d0b2814f2-registry-tls\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.824022 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.824007 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49d6484b-12c8-4554-85cd-fb0d0b2814f2-installation-pull-secrets\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.824188 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.824167 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49d6484b-12c8-4554-85cd-fb0d0b2814f2-image-registry-private-configuration\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.874795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.874742 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49d6484b-12c8-4554-85cd-fb0d0b2814f2-bound-sa-token\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.874925 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.874859 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmsw\" (UniqueName: \"kubernetes.io/projected/49d6484b-12c8-4554-85cd-fb0d0b2814f2-kube-api-access-6tmsw\") pod \"image-registry-6d59d8bb6c-87rbn\" (UID: \"49d6484b-12c8-4554-85cd-fb0d0b2814f2\") " pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:42.944753 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:42.944723 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:43.221905 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.221874 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-fzkgk"] Nov 03 13:04:43.224434 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.224409 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bea2f07b-470d-4c61-8740-989b36f04c36-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2fp88\" (UID: \"bea2f07b-470d-4c61-8740-989b36f04c36\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" Nov 03 13:04:43.224516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.224419 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:04:43.224563 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:43.224509 4541 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Nov 03 13:04:43.224563 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:43.224561 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bea2f07b-470d-4c61-8740-989b36f04c36-tls-certificates podName:bea2f07b-470d-4c61-8740-989b36f04c36 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:44.224549568 +0000 UTC m=+15.538521420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/bea2f07b-470d-4c61-8740-989b36f04c36-tls-certificates") pod "prometheus-operator-admission-webhook-6489d8bf89-2fp88" (UID: "bea2f07b-470d-4c61-8740-989b36f04c36") : secret "prometheus-operator-admission-webhook-tls" not found Nov 03 13:04:43.273946 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.273916 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-h7824\"" Nov 03 13:04:43.273946 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.273930 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 13:04:43.273946 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.273930 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 13:04:43.281660 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.281631 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fzkgk"] Nov 03 13:04:43.325708 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.325674 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcw4s\" (UniqueName: \"kubernetes.io/projected/af914556-87df-4d97-8da1-b049975afeae-kube-api-access-tcw4s\") pod \"downloads-6dd789c8b7-fzkgk\" (UID: \"af914556-87df-4d97-8da1-b049975afeae\") " pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:04:43.325886 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.325823 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af914556-87df-4d97-8da1-b049975afeae-tmp\") pod \"downloads-6dd789c8b7-fzkgk\" (UID: \"af914556-87df-4d97-8da1-b049975afeae\") " pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:04:43.380848 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.380797 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d59d8bb6c-87rbn"] Nov 03 13:04:43.384433 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:43.384409 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d6484b_12c8_4554_85cd_fb0d0b2814f2.slice/crio-6725d726387f60f7826f192944e969b99e9d1bf8791ef733e0188cc5ee253810 WatchSource:0}: Error finding container 6725d726387f60f7826f192944e969b99e9d1bf8791ef733e0188cc5ee253810: Status 404 returned error can't find the container with id 6725d726387f60f7826f192944e969b99e9d1bf8791ef733e0188cc5ee253810 Nov 03 13:04:43.426385 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.426358 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcw4s\" (UniqueName: \"kubernetes.io/projected/af914556-87df-4d97-8da1-b049975afeae-kube-api-access-tcw4s\") pod \"downloads-6dd789c8b7-fzkgk\" (UID: \"af914556-87df-4d97-8da1-b049975afeae\") " pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:04:43.426521 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.426460 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af914556-87df-4d97-8da1-b049975afeae-tmp\") pod \"downloads-6dd789c8b7-fzkgk\" (UID: \"af914556-87df-4d97-8da1-b049975afeae\") " pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:04:43.426829 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.426811 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af914556-87df-4d97-8da1-b049975afeae-tmp\") pod \"downloads-6dd789c8b7-fzkgk\" (UID: \"af914556-87df-4d97-8da1-b049975afeae\") " pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:04:43.483487 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.483400 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcw4s\" (UniqueName: \"kubernetes.io/projected/af914556-87df-4d97-8da1-b049975afeae-kube-api-access-tcw4s\") pod \"downloads-6dd789c8b7-fzkgk\" (UID: \"af914556-87df-4d97-8da1-b049975afeae\") " pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:04:43.534829 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.534754 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:04:43.937383 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.937351 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fzkgk"] Nov 03 13:04:43.940126 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:43.940097 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf914556_87df_4d97_8da1_b049975afeae.slice/crio-036fddda4a1ac515d54d18ce32a6c303f04f4c42e805d938b75bb3162e34c567 WatchSource:0}: Error finding container 036fddda4a1ac515d54d18ce32a6c303f04f4c42e805d938b75bb3162e34c567: Status 404 returned error can't find the container with id 036fddda4a1ac515d54d18ce32a6c303f04f4c42e805d938b75bb3162e34c567 Nov 03 13:04:43.961585 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.961561 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" event={"ID":"49d6484b-12c8-4554-85cd-fb0d0b2814f2","Type":"ContainerStarted","Data":"be00ee8a7e1396504e807ab86b4ffc6cc2b6227e9e36f99498f69ec108db7d72"} Nov 03 13:04:43.961717 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.961603 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" event={"ID":"49d6484b-12c8-4554-85cd-fb0d0b2814f2","Type":"ContainerStarted","Data":"6725d726387f60f7826f192944e969b99e9d1bf8791ef733e0188cc5ee253810"} Nov 03 13:04:43.961717 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.961665 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:04:43.962821 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:43.962799 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fzkgk" event={"ID":"af914556-87df-4d97-8da1-b049975afeae","Type":"ContainerStarted","Data":"036fddda4a1ac515d54d18ce32a6c303f04f4c42e805d938b75bb3162e34c567"} Nov 03 13:04:44.071927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:44.071703 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" podStartSLOduration=2.071689599 podStartE2EDuration="2.071689599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:44.070946819 +0000 UTC m=+15.384918679" watchObservedRunningTime="2026-11-03 13:04:44.071689599 +0000 UTC m=+15.385661503" Nov 03 13:04:44.233531 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:44.233449 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bea2f07b-470d-4c61-8740-989b36f04c36-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2fp88\" (UID: \"bea2f07b-470d-4c61-8740-989b36f04c36\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" Nov 03 13:04:44.235953 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:44.235929 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/bea2f07b-470d-4c61-8740-989b36f04c36-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2fp88\" (UID: \"bea2f07b-470d-4c61-8740-989b36f04c36\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" Nov 03 13:04:44.348208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:44.348165 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" Nov 03 13:04:44.705422 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:44.705386 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88"] Nov 03 13:04:44.707979 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:44.707953 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea2f07b_470d_4c61_8740_989b36f04c36.slice/crio-edfe5fb1c8cc0fcdcf5a80dc652936c74c84c53bb8efb3e210c526aa9072eff7 WatchSource:0}: Error finding container edfe5fb1c8cc0fcdcf5a80dc652936c74c84c53bb8efb3e210c526aa9072eff7: Status 404 returned error can't find the container with id edfe5fb1c8cc0fcdcf5a80dc652936c74c84c53bb8efb3e210c526aa9072eff7 Nov 03 13:04:44.967037 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:44.966968 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" event={"ID":"bea2f07b-470d-4c61-8740-989b36f04c36","Type":"ContainerStarted","Data":"edfe5fb1c8cc0fcdcf5a80dc652936c74c84c53bb8efb3e210c526aa9072eff7"} Nov 03 13:04:45.748205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:45.748162 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:45.751008 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:45.750947 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf9f8413-6613-4db2-9dd1-20742c2af7ed-metrics-certs\") pod \"network-metrics-daemon-r8f65\" (UID: \"cf9f8413-6613-4db2-9dd1-20742c2af7ed\") " pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:45.818083 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:45.818042 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r8f65" Nov 03 13:04:46.194844 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:46.194814 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r8f65"] Nov 03 13:04:46.198758 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:46.198733 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9f8413_6613_4db2_9dd1_20742c2af7ed.slice/crio-0508838039b6f6c6102332f8661d46d385ee8252a452f891461b99329bb0f38e WatchSource:0}: Error finding container 0508838039b6f6c6102332f8661d46d385ee8252a452f891461b99329bb0f38e: Status 404 returned error can't find the container with id 0508838039b6f6c6102332f8661d46d385ee8252a452f891461b99329bb0f38e Nov 03 13:04:46.893712 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:04:46.893677 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:04:46.976473 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:46.976434 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r8f65" event={"ID":"cf9f8413-6613-4db2-9dd1-20742c2af7ed","Type":"ContainerStarted","Data":"0508838039b6f6c6102332f8661d46d385ee8252a452f891461b99329bb0f38e"} Nov 03 13:04:46.978161 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:46.978130 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" event={"ID":"bea2f07b-470d-4c61-8740-989b36f04c36","Type":"ContainerStarted","Data":"c9a937e0f224151f56e09a57860d8f4d64bdd2c6caf7f86cdfb6d282c623d9bf"} Nov 03 13:04:46.978528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:46.978505 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" Nov 03 13:04:46.983571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:46.983544 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" Nov 03 13:04:47.085706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:47.085648 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2fp88" podStartSLOduration=3.485524705 podStartE2EDuration="5.085630237s" totalImagesPullingTime="1.600105532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:47.084989105 +0000 UTC m=+18.398960966" watchObservedRunningTime="2026-11-03 13:04:47.085630237 +0000 UTC m=+18.399602099" Nov 03 13:04:47.985515 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:47.985475 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r8f65" event={"ID":"cf9f8413-6613-4db2-9dd1-20742c2af7ed","Type":"ContainerStarted","Data":"26cae859c619128a6f6b9dbea39d668d35e8ef2d7e0eaa6ffa7acefbbec81c8d"} Nov 03 13:04:48.279873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.279831 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2kn64"] Nov 03 13:04:48.281992 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.281970 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.367635 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.367595 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.367635 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.367637 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlnc8\" (UniqueName: \"kubernetes.io/projected/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-kube-api-access-rlnc8\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.367868 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.367663 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.367868 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.367834 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.379083 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.379053 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 13:04:48.379219 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.379072 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 03 13:04:48.379219 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.379054 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 13:04:48.379219 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.379148 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-cnm6m\"" Nov 03 13:04:48.426120 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.426086 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 13:04:48.426317 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.426293 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 03 13:04:48.427660 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.427639 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2kn64"] Nov 03 13:04:48.468861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.468827 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.469024 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.468873 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.469024 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.468943 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.469024 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.468967 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlnc8\" (UniqueName: \"kubernetes.io/projected/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-kube-api-access-rlnc8\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.469781 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.469748 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.471605 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.471580 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.471693 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.471630 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.523639 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.523610 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlnc8\" (UniqueName: \"kubernetes.io/projected/8c02bcc5-31fc-4914-b0f8-605b0700aa8c-kube-api-access-rlnc8\") pod \"prometheus-operator-9cd89b74c-2kn64\" (UID: \"8c02bcc5-31fc-4914-b0f8-605b0700aa8c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.596648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.596568 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" Nov 03 13:04:48.960205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.960177 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-2kn64"] Nov 03 13:04:48.963569 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:48.963535 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c02bcc5_31fc_4914_b0f8_605b0700aa8c.slice/crio-5ec4ae3f434d6de346e8e41cccfd51bca920dca48e6623f1e3a5c2926fdbda22 WatchSource:0}: Error finding container 5ec4ae3f434d6de346e8e41cccfd51bca920dca48e6623f1e3a5c2926fdbda22: Status 404 returned error can't find the container with id 5ec4ae3f434d6de346e8e41cccfd51bca920dca48e6623f1e3a5c2926fdbda22 Nov 03 13:04:48.993672 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.993634 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" event={"ID":"8c02bcc5-31fc-4914-b0f8-605b0700aa8c","Type":"ContainerStarted","Data":"5ec4ae3f434d6de346e8e41cccfd51bca920dca48e6623f1e3a5c2926fdbda22"} Nov 03 13:04:48.995715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:48.995685 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r8f65" event={"ID":"cf9f8413-6613-4db2-9dd1-20742c2af7ed","Type":"ContainerStarted","Data":"17fe6ec7586ca32102158ccb307375ad877f438653f9335aa691314cabef85a0"} Nov 03 13:04:50.954543 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:50.954461 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b5n5d" Nov 03 13:04:51.006323 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:51.006233 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" event={"ID":"8c02bcc5-31fc-4914-b0f8-605b0700aa8c","Type":"ContainerStarted","Data":"c764203b9b34bcca9bcb5825017a4bab79a515b8a032db5f80897176a1345ce7"} Nov 03 13:04:51.006323 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:51.006299 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" event={"ID":"8c02bcc5-31fc-4914-b0f8-605b0700aa8c","Type":"ContainerStarted","Data":"ff45dba4d7a0c050e74a8bbbd388864be49b23cf7fdca313f9d60f4a5011b174"} Nov 03 13:04:51.200347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:51.200287 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-2kn64" podStartSLOduration=1.446200937 podStartE2EDuration="3.200268574s" totalImagesPullingTime="1.754067637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:51.198909119 +0000 UTC m=+22.512881007" watchObservedRunningTime="2026-11-03 13:04:51.200268574 +0000 UTC m=+22.514240435" Nov 03 13:04:53.815286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.815249 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sdmbx"] Nov 03 13:04:53.817861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.817834 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:53.868560 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.868524 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 13:04:53.868747 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.868644 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 13:04:53.914138 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914106 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-sys\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:53.914319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914161 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-wtmp\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:53.914319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914197 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5rlch\"" Nov 03 13:04:53.914319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914226 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-accelerators-collector-config\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:53.914319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914283 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 13:04:53.914319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914300 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-tls\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:53.914599 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914366 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-textfile\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:53.914599 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914396 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-metrics-client-ca\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:53.914599 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914429 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p5dw\" (UniqueName: \"kubernetes.io/projected/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-kube-api-access-4p5dw\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:53.914599 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914530 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-root\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:53.914599 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:53.914558 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.015413 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015370 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-tls\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.015413 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015419 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-textfile\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.015648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015442 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-metrics-client-ca\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.015648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015472 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p5dw\" (UniqueName: \"kubernetes.io/projected/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-kube-api-access-4p5dw\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.015648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015518 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-root\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.015648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015545 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.015648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015586 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-sys\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.015648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015634 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-wtmp\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.015648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015633 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-root\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.016010 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015657 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-accelerators-collector-config\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.016010 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015752 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-sys\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.016010 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015871 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-textfile\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.016010 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.015948 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-wtmp\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.016588 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.016562 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-metrics-client-ca\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.016670 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.016564 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-accelerators-collector-config\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.018867 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.018843 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-tls\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.018957 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.018916 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.068815 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.068702 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p5dw\" (UniqueName: \"kubernetes.io/projected/01cfefb2-c2f1-4677-a477-e9a7e9afb27d-kube-api-access-4p5dw\") pod \"node-exporter-sdmbx\" (UID: \"01cfefb2-c2f1-4677-a477-e9a7e9afb27d\") " pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.132894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:54.132848 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sdmbx" Nov 03 13:04:54.145356 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:04:54.145325 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cfefb2_c2f1_4677_a477_e9a7e9afb27d.slice/crio-094cdacffb9053221ed39d28ec7fdbb7354cf5b1b04329c8ac77a526409af581 WatchSource:0}: Error finding container 094cdacffb9053221ed39d28ec7fdbb7354cf5b1b04329c8ac77a526409af581: Status 404 returned error can't find the container with id 094cdacffb9053221ed39d28ec7fdbb7354cf5b1b04329c8ac77a526409af581 Nov 03 13:04:55.025086 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:55.025032 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sdmbx" event={"ID":"01cfefb2-c2f1-4677-a477-e9a7e9afb27d","Type":"ContainerStarted","Data":"094cdacffb9053221ed39d28ec7fdbb7354cf5b1b04329c8ac77a526409af581"} Nov 03 13:04:56.030123 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:56.030082 4541 generic.go:356] "Generic (PLEG): container finished" podID="01cfefb2-c2f1-4677-a477-e9a7e9afb27d" containerID="cad9b67c5934d9ceb685b5617d032fc9c14966d72e2eef7a1cd36d1ef05e2957" exitCode=0 Nov 03 13:04:56.030699 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:56.030173 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sdmbx" event={"ID":"01cfefb2-c2f1-4677-a477-e9a7e9afb27d","Type":"ContainerDied","Data":"cad9b67c5934d9ceb685b5617d032fc9c14966d72e2eef7a1cd36d1ef05e2957"} Nov 03 13:04:57.036439 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:57.036399 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sdmbx" event={"ID":"01cfefb2-c2f1-4677-a477-e9a7e9afb27d","Type":"ContainerStarted","Data":"a0dfda47061b371091c24d689ec761c2d5ae9b1a24b5deec6c0652b9698f0777"} Nov 03 13:04:57.036888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:57.036464 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sdmbx" event={"ID":"01cfefb2-c2f1-4677-a477-e9a7e9afb27d","Type":"ContainerStarted","Data":"9b466eb06155416c0019ccda093560f7f1cdd2950f52a16bf0b7e9ef7501516d"} Nov 03 13:04:57.196513 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:57.196445 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sdmbx" podStartSLOduration=2.659374582 podStartE2EDuration="4.196425354s" totalImagesPullingTime="1.461426653s" totalInitContainerRuntime="75.624119ms" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:57.195706699 +0000 UTC m=+28.509678556" watchObservedRunningTime="2026-11-03 13:04:57.196425354 +0000 UTC m=+28.510397214" Nov 03 13:04:59.730034 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:04:59.730004 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hfbh8" Nov 03 13:05:01.052404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:01.052369 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fzkgk" event={"ID":"af914556-87df-4d97-8da1-b049975afeae","Type":"ContainerStarted","Data":"37a15ec6065281a8362b41f5aaff7ed5d91ae2f9614cd10e0957f1e85eb95127"} Nov 03 13:05:01.052815 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:01.052589 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:05:01.054035 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:01.054011 4541 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-fzkgk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" start-of-body= Nov 03 13:05:01.054141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:01.054061 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-fzkgk" podUID="af914556-87df-4d97-8da1-b049975afeae" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.14:8080/\": dial tcp 10.133.0.14:8080: connect: connection refused" Nov 03 13:05:01.247490 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:01.247427 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-fzkgk" podStartSLOduration=1.3503766050000001 podStartE2EDuration="18.24740439s" totalImagesPullingTime="16.897027785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:01.208344159 +0000 UTC m=+32.522316041" watchObservedRunningTime="2026-11-03 13:05:01.24740439 +0000 UTC m=+32.561376251" Nov 03 13:05:02.070569 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:02.070534 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-fzkgk" Nov 03 13:05:04.971572 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:04.971543 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d59d8bb6c-87rbn" Nov 03 13:05:16.887744 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:05:16.887656 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:05:46.888322 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:05:46.888283 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:05:53.229826 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:53.229795 4541 generic.go:356] "Generic (PLEG): container finished" podID="f31d9405-aac0-42bd-8c9a-977b6328bc65" containerID="7a585693c91b19ab462d6691b9069a5cc47df86570d8032fcfb8006167ff9171" exitCode=0 Nov 03 13:05:53.230234 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:53.229866 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" event={"ID":"f31d9405-aac0-42bd-8c9a-977b6328bc65","Type":"ContainerDied","Data":"7a585693c91b19ab462d6691b9069a5cc47df86570d8032fcfb8006167ff9171"} Nov 03 13:05:54.369354 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.369328 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:05:54.473328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.473284 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle\") pod \"f31d9405-aac0-42bd-8c9a-977b6328bc65\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " Nov 03 13:05:54.473520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.473339 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path\") pod \"f31d9405-aac0-42bd-8c9a-977b6328bc65\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " Nov 03 13:05:54.473520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.473368 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle\") pod \"f31d9405-aac0-42bd-8c9a-977b6328bc65\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " Nov 03 13:05:54.473520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.473406 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h\" (UniqueName: \"kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h\") pod \"f31d9405-aac0-42bd-8c9a-977b6328bc65\" (UID: \"f31d9405-aac0-42bd-8c9a-977b6328bc65\") " Nov 03 13:05:54.473662 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.473636 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path" pod "f31d9405-aac0-42bd-8c9a-977b6328bc65" (UID: "f31d9405-aac0-42bd-8c9a-977b6328bc65"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:05:54.473719 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.473693 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle" pod "f31d9405-aac0-42bd-8c9a-977b6328bc65" (UID: "f31d9405-aac0-42bd-8c9a-977b6328bc65"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:54.473754 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.473728 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle" pod "f31d9405-aac0-42bd-8c9a-977b6328bc65" (UID: "f31d9405-aac0-42bd-8c9a-977b6328bc65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:54.475636 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.475608 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h" pod "f31d9405-aac0-42bd-8c9a-977b6328bc65" (UID: "f31d9405-aac0-42bd-8c9a-977b6328bc65"). InnerVolumeSpecName "kube-api-access-q5l9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:05:54.574399 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.574311 4541 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-service-ca-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:54.574399 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.574348 4541 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f31d9405-aac0-42bd-8c9a-977b6328bc65-archives-path\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:54.574399 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.574358 4541 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f31d9405-aac0-42bd-8c9a-977b6328bc65-trusted-ca-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:54.574399 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:54.574367 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5l9h\" (UniqueName: \"kubernetes.io/projected/f31d9405-aac0-42bd-8c9a-977b6328bc65-kube-api-access-q5l9h\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:55.237833 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:55.237798 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" event={"ID":"f31d9405-aac0-42bd-8c9a-977b6328bc65","Type":"ContainerDied","Data":"81a0b663aae2a9e50f52cef108bf7d7f29c256ce7a85f067f346918dcd326d0b"} Nov 03 13:05:55.237833 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:55.237830 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-9xmlk-2g7w7" Nov 03 13:05:55.238038 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:55.237844 4541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81a0b663aae2a9e50f52cef108bf7d7f29c256ce7a85f067f346918dcd326d0b" Nov 03 13:05:59.526641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.526609 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-zgxvf-w87tq"] Nov 03 13:05:59.527026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.526895 4541 cpu_manager.go:414] "containerMap: removing container" podUID="f31d9405-aac0-42bd-8c9a-977b6328bc65" containerName="insights-gathering" Nov 03 13:05:59.527026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.526907 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31d9405-aac0-42bd-8c9a-977b6328bc65" containerName="insights-gathering" Nov 03 13:05:59.527026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.526960 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31d9405-aac0-42bd-8c9a-977b6328bc65" containerName="insights-gathering" Nov 03 13:05:59.528840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.528822 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.613803 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.613748 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f81d8b37-17c7-4cc0-a834-e72782465c54-archives-path\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.613974 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.613815 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-service-ca-bundle\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.613974 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.613900 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-trusted-ca-bundle\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.613974 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.613927 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qx8\" (UniqueName: \"kubernetes.io/projected/f81d8b37-17c7-4cc0-a834-e72782465c54-kube-api-access-t9qx8\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.625520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.625454 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 13:05:59.672612 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.672577 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-697z8\"" Nov 03 13:05:59.714396 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.714361 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-service-ca-bundle\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.714553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.714426 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-trusted-ca-bundle\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.714553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.714512 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qx8\" (UniqueName: \"kubernetes.io/projected/f81d8b37-17c7-4cc0-a834-e72782465c54-kube-api-access-t9qx8\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.714655 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.714581 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f81d8b37-17c7-4cc0-a834-e72782465c54-archives-path\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.714896 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.714879 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f81d8b37-17c7-4cc0-a834-e72782465c54-archives-path\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.715066 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.715048 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-service-ca-bundle\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.720608 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.720590 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 13:05:59.721553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.721535 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zgxvf-w87tq"] Nov 03 13:05:59.725450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.725426 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-trusted-ca-bundle\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.769373 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.769343 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qx8\" (UniqueName: \"kubernetes.io/projected/f81d8b37-17c7-4cc0-a834-e72782465c54-kube-api-access-t9qx8\") pod \"periodic-gathering-zgxvf-w87tq\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:05:59.838908 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:05:59.838812 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:06:00.195124 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:06:00.195096 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-zgxvf-w87tq"] Nov 03 13:06:00.198012 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:06:00.197983 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81d8b37_17c7_4cc0_a834_e72782465c54.slice/crio-c3475318ca76eacbab3caab9ed69b5211924b1fadb08afef82071bab66e2cdc0 WatchSource:0}: Error finding container c3475318ca76eacbab3caab9ed69b5211924b1fadb08afef82071bab66e2cdc0: Status 404 returned error can't find the container with id c3475318ca76eacbab3caab9ed69b5211924b1fadb08afef82071bab66e2cdc0 Nov 03 13:06:00.255800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:06:00.255754 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zgxvf-w87tq" event={"ID":"f81d8b37-17c7-4cc0-a834-e72782465c54","Type":"ContainerStarted","Data":"f7c2dc9c6a7fd7798e32148438386fe4c73ab1113c5b1db0a749639084d3d9d0"} Nov 03 13:06:00.255932 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:06:00.255832 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zgxvf-w87tq" event={"ID":"f81d8b37-17c7-4cc0-a834-e72782465c54","Type":"ContainerStarted","Data":"c3475318ca76eacbab3caab9ed69b5211924b1fadb08afef82071bab66e2cdc0"} Nov 03 13:06:00.363698 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:06:00.363644 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-zgxvf-w87tq" podStartSLOduration=1.363628436 podStartE2EDuration="1.363628436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:06:00.362547416 +0000 UTC m=+91.676519276" watchObservedRunningTime="2026-11-03 13:06:00.363628436 +0000 UTC m=+91.677600331" Nov 03 13:06:16.887062 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:06:16.887026 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:06:46.889098 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:06:46.889052 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:07:16.889040 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:07:16.889000 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:07:27.554885 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:27.554850 4541 generic.go:356] "Generic (PLEG): container finished" podID="f81d8b37-17c7-4cc0-a834-e72782465c54" containerID="f7c2dc9c6a7fd7798e32148438386fe4c73ab1113c5b1db0a749639084d3d9d0" exitCode=0 Nov 03 13:07:27.555294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:27.554907 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zgxvf-w87tq" event={"ID":"f81d8b37-17c7-4cc0-a834-e72782465c54","Type":"ContainerDied","Data":"f7c2dc9c6a7fd7798e32148438386fe4c73ab1113c5b1db0a749639084d3d9d0"} Nov 03 13:07:28.680521 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.680492 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:07:28.745546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.745509 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-trusted-ca-bundle\") pod \"f81d8b37-17c7-4cc0-a834-e72782465c54\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " Nov 03 13:07:28.745745 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.745563 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f81d8b37-17c7-4cc0-a834-e72782465c54-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f81d8b37-17c7-4cc0-a834-e72782465c54-archives-path\") pod \"f81d8b37-17c7-4cc0-a834-e72782465c54\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " Nov 03 13:07:28.745745 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.745615 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-service-ca-bundle\") pod \"f81d8b37-17c7-4cc0-a834-e72782465c54\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " Nov 03 13:07:28.745745 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.745634 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f81d8b37-17c7-4cc0-a834-e72782465c54-kube-api-access-t9qx8\" (UniqueName: \"kubernetes.io/projected/f81d8b37-17c7-4cc0-a834-e72782465c54-kube-api-access-t9qx8\") pod \"f81d8b37-17c7-4cc0-a834-e72782465c54\" (UID: \"f81d8b37-17c7-4cc0-a834-e72782465c54\") " Nov 03 13:07:28.745985 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.745957 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f81d8b37-17c7-4cc0-a834-e72782465c54-archives-path" pod "f81d8b37-17c7-4cc0-a834-e72782465c54" (UID: "f81d8b37-17c7-4cc0-a834-e72782465c54"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:07:28.745985 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.745972 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-service-ca-bundle" pod "f81d8b37-17c7-4cc0-a834-e72782465c54" (UID: "f81d8b37-17c7-4cc0-a834-e72782465c54"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:07:28.746069 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.745994 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-trusted-ca-bundle" pod "f81d8b37-17c7-4cc0-a834-e72782465c54" (UID: "f81d8b37-17c7-4cc0-a834-e72782465c54"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:07:28.747642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.747624 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f81d8b37-17c7-4cc0-a834-e72782465c54-kube-api-access-t9qx8" pod "f81d8b37-17c7-4cc0-a834-e72782465c54" (UID: "f81d8b37-17c7-4cc0-a834-e72782465c54"). InnerVolumeSpecName "kube-api-access-t9qx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:07:28.846109 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.846017 4541 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-service-ca-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:07:28.846109 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.846057 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9qx8\" (UniqueName: \"kubernetes.io/projected/f81d8b37-17c7-4cc0-a834-e72782465c54-kube-api-access-t9qx8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:07:28.846109 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.846069 4541 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81d8b37-17c7-4cc0-a834-e72782465c54-trusted-ca-bundle\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:07:28.846109 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:28.846080 4541 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f81d8b37-17c7-4cc0-a834-e72782465c54-archives-path\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:07:29.561640 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:29.561605 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-zgxvf-w87tq" event={"ID":"f81d8b37-17c7-4cc0-a834-e72782465c54","Type":"ContainerDied","Data":"c3475318ca76eacbab3caab9ed69b5211924b1fadb08afef82071bab66e2cdc0"} Nov 03 13:07:29.561640 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:29.561639 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-zgxvf-w87tq" Nov 03 13:07:29.561868 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:07:29.561652 4541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3475318ca76eacbab3caab9ed69b5211924b1fadb08afef82071bab66e2cdc0" Nov 03 13:07:46.888558 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:07:46.888518 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:08:16.888503 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:08:16.888427 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:08:46.889936 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:08:46.889894 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:08:46.983147 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:46.983114 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf"] Nov 03 13:08:46.983365 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:46.983353 4541 cpu_manager.go:414] "containerMap: removing container" podUID="f81d8b37-17c7-4cc0-a834-e72782465c54" containerName="insights-gathering" Nov 03 13:08:46.983408 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:46.983367 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f81d8b37-17c7-4cc0-a834-e72782465c54" containerName="insights-gathering" Nov 03 13:08:46.983441 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:46.983430 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="f81d8b37-17c7-4cc0-a834-e72782465c54" containerName="insights-gathering" Nov 03 13:08:46.984961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:46.984944 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.081951 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.081914 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bjkx\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-kube-api-access-4bjkx\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.081951 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.081948 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.082139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.081972 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.082139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.082077 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.082139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.082111 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.082139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.082132 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.082259 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.082209 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7630315-ae07-42ed-92ff-f574651ce623-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.083114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.083090 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 13:08:47.083114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.083107 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 13:08:47.083286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.083108 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ffbgv\"" Nov 03 13:08:47.083286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.083123 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 13:08:47.083358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.083327 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 13:08:47.083584 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.083567 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 13:08:47.084607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.084585 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf"] Nov 03 13:08:47.136442 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.136409 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 13:08:47.182810 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.182762 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7630315-ae07-42ed-92ff-f574651ce623-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.182954 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.182820 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bjkx\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-kube-api-access-4bjkx\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.182954 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.182923 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.182954 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.182950 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.183114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.182971 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.183114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.183001 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.183114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.183031 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.183665 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.183633 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.185198 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.185179 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7630315-ae07-42ed-92ff-f574651ce623-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.185382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.185362 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.185687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.185666 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.185811 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.185791 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.237286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.237245 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.237426 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.237374 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bjkx\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-kube-api-access-4bjkx\") pod \"istiod-openshift-gateway-7876bfcb74-lp5bf\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.294968 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.294935 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:47.705432 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.704258 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf"] Nov 03 13:08:47.707114 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:08:47.707088 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7630315_ae07_42ed_92ff_f574651ce623.slice/crio-111abd2c75f09017ac0700cfc0a30cbaac02f1c13193b11c3f0220a5fd5160f8 WatchSource:0}: Error finding container 111abd2c75f09017ac0700cfc0a30cbaac02f1c13193b11c3f0220a5fd5160f8: Status 404 returned error can't find the container with id 111abd2c75f09017ac0700cfc0a30cbaac02f1c13193b11c3f0220a5fd5160f8 Nov 03 13:08:47.814368 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:47.814336 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" event={"ID":"c7630315-ae07-42ed-92ff-f574651ce623","Type":"ContainerStarted","Data":"111abd2c75f09017ac0700cfc0a30cbaac02f1c13193b11c3f0220a5fd5160f8"} Nov 03 13:08:51.495083 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:51.495048 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:08:51.495370 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:51.495115 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:08:51.829751 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:51.829652 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" event={"ID":"c7630315-ae07-42ed-92ff-f574651ce623","Type":"ContainerStarted","Data":"0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599"} Nov 03 13:08:51.829751 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:51.829724 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:08:52.035499 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:52.035435 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" podStartSLOduration=2.250087504 podStartE2EDuration="6.035415509s" totalImagesPullingTime="3.785328005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:51.987742675 +0000 UTC m=+263.301714536" watchObservedRunningTime="2026-11-03 13:08:52.035415509 +0000 UTC m=+263.349387370" Nov 03 13:08:52.834475 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:52.834442 4541 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-lp5bf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:08:52.834857 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:52.834502 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" podUID="c7630315-ae07-42ed-92ff-f574651ce623" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:08:55.834351 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:08:55.834317 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:09:16.888327 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:09:16.888285 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:09:28.870724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:28.870698 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:09:28.873038 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:28.873011 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:09:30.599098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.599060 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp"] Nov 03 13:09:30.608528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.608493 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.705955 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.705922 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/84ead5e3-58db-45e9-98e1-34832ffccfcf-manifests\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.705955 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.705963 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/84ead5e3-58db-45e9-98e1-34832ffccfcf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.706174 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.705993 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84ead5e3-58db-45e9-98e1-34832ffccfcf-tmp\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.706174 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.706048 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94q8\" (UniqueName: \"kubernetes.io/projected/84ead5e3-58db-45e9-98e1-34832ffccfcf-kube-api-access-t94q8\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.706174 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.706135 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84ead5e3-58db-45e9-98e1-34832ffccfcf-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.706292 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.706180 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/84ead5e3-58db-45e9-98e1-34832ffccfcf-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.717835 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.717803 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 13:09:30.717968 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.717934 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 13:09:30.718026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.717965 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 13:09:30.718026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.717967 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-clsch\"" Nov 03 13:09:30.718026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.717986 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 13:09:30.718026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.717997 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 13:09:30.740786 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.740734 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp"] Nov 03 13:09:30.807128 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.807096 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/84ead5e3-58db-45e9-98e1-34832ffccfcf-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.807320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.807146 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/84ead5e3-58db-45e9-98e1-34832ffccfcf-manifests\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.807320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.807165 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/84ead5e3-58db-45e9-98e1-34832ffccfcf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.807320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.807192 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84ead5e3-58db-45e9-98e1-34832ffccfcf-tmp\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.807320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.807209 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94q8\" (UniqueName: \"kubernetes.io/projected/84ead5e3-58db-45e9-98e1-34832ffccfcf-kube-api-access-t94q8\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.807320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.807231 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84ead5e3-58db-45e9-98e1-34832ffccfcf-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.807618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.807589 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/84ead5e3-58db-45e9-98e1-34832ffccfcf-manifests\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.809729 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.809706 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/84ead5e3-58db-45e9-98e1-34832ffccfcf-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.809864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.809745 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/84ead5e3-58db-45e9-98e1-34832ffccfcf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.809864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.809854 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84ead5e3-58db-45e9-98e1-34832ffccfcf-tmp\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.810106 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.810086 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/84ead5e3-58db-45e9-98e1-34832ffccfcf-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.861993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.861916 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94q8\" (UniqueName: \"kubernetes.io/projected/84ead5e3-58db-45e9-98e1-34832ffccfcf-kube-api-access-t94q8\") pod \"opendatahub-operator-controller-manager-5b7574c576-hwnhp\" (UID: \"84ead5e3-58db-45e9-98e1-34832ffccfcf\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:30.923028 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:30.922990 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:31.399568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:31.399533 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp"] Nov 03 13:09:31.448356 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:09:31.448327 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ead5e3_58db_45e9_98e1_34832ffccfcf.slice/crio-b73c3caca8be34a59f21168f2be3e559d666a702b48bc9e379df17ddf8d760b0 WatchSource:0}: Error finding container b73c3caca8be34a59f21168f2be3e559d666a702b48bc9e379df17ddf8d760b0: Status 404 returned error can't find the container with id b73c3caca8be34a59f21168f2be3e559d666a702b48bc9e379df17ddf8d760b0 Nov 03 13:09:31.449953 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:31.449931 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:09:31.960236 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:31.960198 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" event={"ID":"84ead5e3-58db-45e9-98e1-34832ffccfcf","Type":"ContainerStarted","Data":"b73c3caca8be34a59f21168f2be3e559d666a702b48bc9e379df17ddf8d760b0"} Nov 03 13:09:35.977614 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:35.977572 4541 generic.go:356] "Generic (PLEG): container finished" podID="84ead5e3-58db-45e9-98e1-34832ffccfcf" containerID="36dc99859cb46c21ef25db098aa9b34796f754300879bb66a7b57ae24368226a" exitCode=0 Nov 03 13:09:35.978058 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:35.977656 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" event={"ID":"84ead5e3-58db-45e9-98e1-34832ffccfcf","Type":"ContainerDied","Data":"36dc99859cb46c21ef25db098aa9b34796f754300879bb66a7b57ae24368226a"} Nov 03 13:09:36.983747 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:36.983711 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" event={"ID":"84ead5e3-58db-45e9-98e1-34832ffccfcf","Type":"ContainerStarted","Data":"6f0b40a5460ba87710a51d5446b74b31b6aaf572c8265b79c9ab3fb2e07fad7b"} Nov 03 13:09:36.984135 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:36.983823 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:09:37.209855 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:37.209761 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" podStartSLOduration=2.310875273 podStartE2EDuration="7.209748329s" totalImagesPullingTime="4.400143061s" totalInitContainerRuntime="498.729995ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:37.209338829 +0000 UTC m=+308.523310689" watchObservedRunningTime="2026-11-03 13:09:37.209748329 +0000 UTC m=+308.523720189" Nov 03 13:09:46.888229 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:09:46.888185 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:09:47.989915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:09:47.989881 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-hwnhp" Nov 03 13:10:10.877741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:10.877705 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77875f86b8-n5psr"] Nov 03 13:10:10.880980 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:10.880955 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:10.956643 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:10.956612 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77875f86b8-n5psr"] Nov 03 13:10:10.977604 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:10.977562 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 13:10:10.977604 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:10.977577 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p6lgj\"" Nov 03 13:10:10.977604 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:10.977609 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 13:10:11.021273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.021238 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c2eb10-7803-4a98-9859-1ab515a46bce-tls-certs\") pod \"kube-auth-proxy-77875f86b8-n5psr\" (UID: \"f8c2eb10-7803-4a98-9859-1ab515a46bce\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.021451 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.021302 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8j6\" (UniqueName: \"kubernetes.io/projected/f8c2eb10-7803-4a98-9859-1ab515a46bce-kube-api-access-hz8j6\") pod \"kube-auth-proxy-77875f86b8-n5psr\" (UID: \"f8c2eb10-7803-4a98-9859-1ab515a46bce\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.021451 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.021390 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8c2eb10-7803-4a98-9859-1ab515a46bce-tmp\") pod \"kube-auth-proxy-77875f86b8-n5psr\" (UID: \"f8c2eb10-7803-4a98-9859-1ab515a46bce\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.121974 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.121935 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c2eb10-7803-4a98-9859-1ab515a46bce-tls-certs\") pod \"kube-auth-proxy-77875f86b8-n5psr\" (UID: \"f8c2eb10-7803-4a98-9859-1ab515a46bce\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.122159 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.121994 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8j6\" (UniqueName: \"kubernetes.io/projected/f8c2eb10-7803-4a98-9859-1ab515a46bce-kube-api-access-hz8j6\") pod \"kube-auth-proxy-77875f86b8-n5psr\" (UID: \"f8c2eb10-7803-4a98-9859-1ab515a46bce\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.122159 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.122021 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8c2eb10-7803-4a98-9859-1ab515a46bce-tmp\") pod \"kube-auth-proxy-77875f86b8-n5psr\" (UID: \"f8c2eb10-7803-4a98-9859-1ab515a46bce\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.124265 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.124233 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f8c2eb10-7803-4a98-9859-1ab515a46bce-tmp\") pod \"kube-auth-proxy-77875f86b8-n5psr\" (UID: \"f8c2eb10-7803-4a98-9859-1ab515a46bce\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.124502 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.124482 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8c2eb10-7803-4a98-9859-1ab515a46bce-tls-certs\") pod \"kube-auth-proxy-77875f86b8-n5psr\" (UID: \"f8c2eb10-7803-4a98-9859-1ab515a46bce\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.176445 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.176411 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8j6\" (UniqueName: \"kubernetes.io/projected/f8c2eb10-7803-4a98-9859-1ab515a46bce-kube-api-access-hz8j6\") pod \"kube-auth-proxy-77875f86b8-n5psr\" (UID: \"f8c2eb10-7803-4a98-9859-1ab515a46bce\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.193263 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.193239 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" Nov 03 13:10:11.554466 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:11.554304 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77875f86b8-n5psr"] Nov 03 13:10:11.557033 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:10:11.556996 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c2eb10_7803_4a98_9859_1ab515a46bce.slice/crio-bad748508fe8fbd0e9112aab1d12a19717ee7789d236eff0b04b54f06aa11c1d WatchSource:0}: Error finding container bad748508fe8fbd0e9112aab1d12a19717ee7789d236eff0b04b54f06aa11c1d: Status 404 returned error can't find the container with id bad748508fe8fbd0e9112aab1d12a19717ee7789d236eff0b04b54f06aa11c1d Nov 03 13:10:12.103606 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:12.103567 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" event={"ID":"f8c2eb10-7803-4a98-9859-1ab515a46bce","Type":"ContainerStarted","Data":"bad748508fe8fbd0e9112aab1d12a19717ee7789d236eff0b04b54f06aa11c1d"} Nov 03 13:10:15.116477 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:15.116385 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" event={"ID":"f8c2eb10-7803-4a98-9859-1ab515a46bce","Type":"ContainerStarted","Data":"a1c56b2980f70dd485c7d5ab7a30bba7840e076c923a20ddb19519c3726891b1"} Nov 03 13:10:15.226046 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:15.225992 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77875f86b8-n5psr" podStartSLOduration=2.044867665 podStartE2EDuration="5.225977128s" totalImagesPullingTime="3.181109463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:15.225117491 +0000 UTC m=+346.539089350" watchObservedRunningTime="2026-11-03 13:10:15.225977128 +0000 UTC m=+346.539948987" Nov 03 13:10:16.888425 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:10:16.888389 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:10:27.801839 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.801802 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g"] Nov 03 13:10:27.805307 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.805283 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:27.904997 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.904956 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:10:27.905185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.905037 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 03 13:10:27.905185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.905106 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 03 13:10:27.905185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.905151 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 03 13:10:27.905185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.905162 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 13:10:27.905352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.905269 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-sz56b\"" Nov 03 13:10:27.906234 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.906214 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g"] Nov 03 13:10:27.948639 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.948603 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5wn6\" (UniqueName: \"kubernetes.io/projected/1c6a2219-775d-40ea-831a-574dc290c5de-kube-api-access-k5wn6\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:27.948859 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.948661 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1c6a2219-775d-40ea-831a-574dc290c5de-manager-config\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:27.948859 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.948812 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6a2219-775d-40ea-831a-574dc290c5de-metrics-certs\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:27.948978 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:27.948863 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c6a2219-775d-40ea-831a-574dc290c5de-cert\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.049906 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.049865 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1c6a2219-775d-40ea-831a-574dc290c5de-manager-config\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.050088 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.049918 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6a2219-775d-40ea-831a-574dc290c5de-metrics-certs\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.050088 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.049951 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c6a2219-775d-40ea-831a-574dc290c5de-cert\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.050088 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.049982 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5wn6\" (UniqueName: \"kubernetes.io/projected/1c6a2219-775d-40ea-831a-574dc290c5de-kube-api-access-k5wn6\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.050669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.050649 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1c6a2219-775d-40ea-831a-574dc290c5de-manager-config\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.052576 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.052519 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c6a2219-775d-40ea-831a-574dc290c5de-cert\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.052667 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.052597 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c6a2219-775d-40ea-831a-574dc290c5de-metrics-certs\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.105460 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.105419 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5wn6\" (UniqueName: \"kubernetes.io/projected/1c6a2219-775d-40ea-831a-574dc290c5de-kube-api-access-k5wn6\") pod \"jobset-controller-manager-75f97dcd84-2vl6g\" (UID: \"1c6a2219-775d-40ea-831a-574dc290c5de\") " pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.117252 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.117225 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:28.478209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:28.478177 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g"] Nov 03 13:10:28.481501 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:10:28.481469 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c6a2219_775d_40ea_831a_574dc290c5de.slice/crio-ad2265b9ad3ec8cc0fe3682fc8526c21af8d64f790281cc7aeb4b17c40229f55 WatchSource:0}: Error finding container ad2265b9ad3ec8cc0fe3682fc8526c21af8d64f790281cc7aeb4b17c40229f55: Status 404 returned error can't find the container with id ad2265b9ad3ec8cc0fe3682fc8526c21af8d64f790281cc7aeb4b17c40229f55 Nov 03 13:10:29.172583 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:29.172542 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" event={"ID":"1c6a2219-775d-40ea-831a-574dc290c5de","Type":"ContainerStarted","Data":"ad2265b9ad3ec8cc0fe3682fc8526c21af8d64f790281cc7aeb4b17c40229f55"} Nov 03 13:10:33.191075 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:33.191036 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" event={"ID":"1c6a2219-775d-40ea-831a-574dc290c5de","Type":"ContainerStarted","Data":"729c19f3bcded9194b03bfe02e29365e1b325ec3d0217bc29f30140cf1562681"} Nov 03 13:10:33.191492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:33.191136 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:33.300353 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:33.300290 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" podStartSLOduration=2.240194926 podStartE2EDuration="6.300277265s" totalImagesPullingTime="4.060082339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:33.299923116 +0000 UTC m=+364.613894976" watchObservedRunningTime="2026-11-03 13:10:33.300277265 +0000 UTC m=+364.614249119" Nov 03 13:10:44.200001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:10:44.199969 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-75f97dcd84-2vl6g" Nov 03 13:10:46.887939 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:10:46.887903 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:11:16.889104 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:11:16.889023 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:11:44.016189 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.016147 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf"] Nov 03 13:11:44.016604 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.016419 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" podUID="c7630315-ae07-42ed-92ff-f574651ce623" containerName="discovery" containerID="cri-o://0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599" gracePeriod=30 Nov 03 13:11:44.260432 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.260407 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:11:44.443608 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.443570 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-kube-api-access-4bjkx\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-kube-api-access-4bjkx\") pod \"c7630315-ae07-42ed-92ff-f574651ce623\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " Nov 03 13:11:44.443809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.443627 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7630315-ae07-42ed-92ff-f574651ce623-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7630315-ae07-42ed-92ff-f574651ce623-local-certs\") pod \"c7630315-ae07-42ed-92ff-f574651ce623\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " Nov 03 13:11:44.443809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.443708 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-cacerts\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-cacerts\") pod \"c7630315-ae07-42ed-92ff-f574651ce623\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " Nov 03 13:11:44.443809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.443755 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-ca-configmap\") pod \"c7630315-ae07-42ed-92ff-f574651ce623\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " Nov 03 13:11:44.443809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.443804 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-istio-token\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-istio-token\") pod \"c7630315-ae07-42ed-92ff-f574651ce623\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " Nov 03 13:11:44.444021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.443827 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-dns-cert\") pod \"c7630315-ae07-42ed-92ff-f574651ce623\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " Nov 03 13:11:44.444021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.443855 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-kubeconfig\") pod \"c7630315-ae07-42ed-92ff-f574651ce623\" (UID: \"c7630315-ae07-42ed-92ff-f574651ce623\") " Nov 03 13:11:44.444197 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.444159 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-ca-configmap" pod "c7630315-ae07-42ed-92ff-f574651ce623" (UID: "c7630315-ae07-42ed-92ff-f574651ce623"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:11:44.446106 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.446079 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7630315-ae07-42ed-92ff-f574651ce623-local-certs" pod "c7630315-ae07-42ed-92ff-f574651ce623" (UID: "c7630315-ae07-42ed-92ff-f574651ce623"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:11:44.446245 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.446208 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-kube-api-access-4bjkx" pod "c7630315-ae07-42ed-92ff-f574651ce623" (UID: "c7630315-ae07-42ed-92ff-f574651ce623"). InnerVolumeSpecName "kube-api-access-4bjkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:11:44.446358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.446262 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-cacerts" pod "c7630315-ae07-42ed-92ff-f574651ce623" (UID: "c7630315-ae07-42ed-92ff-f574651ce623"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:11:44.446358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.446291 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-istio-token" pod "c7630315-ae07-42ed-92ff-f574651ce623" (UID: "c7630315-ae07-42ed-92ff-f574651ce623"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:11:44.446436 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.446398 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-dns-cert" pod "c7630315-ae07-42ed-92ff-f574651ce623" (UID: "c7630315-ae07-42ed-92ff-f574651ce623"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:11:44.446470 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.446452 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-kubeconfig" pod "c7630315-ae07-42ed-92ff-f574651ce623" (UID: "c7630315-ae07-42ed-92ff-f574651ce623"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:11:44.460882 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.460857 4541 generic.go:356] "Generic (PLEG): container finished" podID="c7630315-ae07-42ed-92ff-f574651ce623" containerID="0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599" exitCode=0 Nov 03 13:11:44.460996 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.460899 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" Nov 03 13:11:44.460996 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.460914 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" event={"ID":"c7630315-ae07-42ed-92ff-f574651ce623","Type":"ContainerDied","Data":"0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599"} Nov 03 13:11:44.460996 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.460954 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf" event={"ID":"c7630315-ae07-42ed-92ff-f574651ce623","Type":"ContainerDied","Data":"111abd2c75f09017ac0700cfc0a30cbaac02f1c13193b11c3f0220a5fd5160f8"} Nov 03 13:11:44.460996 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.460984 4541 scope.go:122] "RemoveContainer" containerID="0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599" Nov 03 13:11:44.470050 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.470032 4541 scope.go:122] "RemoveContainer" containerID="0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599" Nov 03 13:11:44.470287 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:11:44.470268 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599\": container with ID starting with 0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599 not found: ID does not exist" containerID="0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599" Nov 03 13:11:44.470331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.470296 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599"} err="failed to get container status \"0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599\": rpc error: code = NotFound desc = could not find container \"0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599\": container with ID starting with 0af199ef6424a59a5b15d866556adba8e871fc6bdae2dfe93ad6bcdea6261599 not found: ID does not exist" Nov 03 13:11:44.545036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.544998 4541 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-cacerts\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:44.545036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.545030 4541 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-ca-configmap\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:44.545036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.545040 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:44.545282 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.545050 4541 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-csr-dns-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:44.545282 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.545058 4541 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c7630315-ae07-42ed-92ff-f574651ce623-istio-kubeconfig\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:44.545282 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.545067 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bjkx\" (UniqueName: \"kubernetes.io/projected/c7630315-ae07-42ed-92ff-f574651ce623-kube-api-access-4bjkx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:44.545282 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.545075 4541 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7630315-ae07-42ed-92ff-f574651ce623-local-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:44.616063 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.616031 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf"] Nov 03 13:11:44.621009 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.620985 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-lp5bf"] Nov 03 13:11:44.865934 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:44.865860 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7630315-ae07-42ed-92ff-f574651ce623" path="/var/lib/kubelet/pods/c7630315-ae07-42ed-92ff-f574651ce623/volumes" Nov 03 13:11:46.890904 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:11:46.890870 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:11:52.348074 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.348032 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68c645947f-qmmh7"] Nov 03 13:11:52.348569 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.348325 4541 cpu_manager.go:414] "containerMap: removing container" podUID="c7630315-ae07-42ed-92ff-f574651ce623" containerName="discovery" Nov 03 13:11:52.348569 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.348336 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7630315-ae07-42ed-92ff-f574651ce623" containerName="discovery" Nov 03 13:11:52.348569 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.348400 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7630315-ae07-42ed-92ff-f574651ce623" containerName="discovery" Nov 03 13:11:52.351073 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.351053 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.402421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.402389 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:11:52.402421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.402411 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:11:52.405929 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.405902 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-metrics-certs\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.405929 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.405943 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-maas-api-tls\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.406122 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.405976 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45nbg\" (UniqueName: \"kubernetes.io/projected/ed3a9bb7-0493-4eef-b193-035ef927d453-kube-api-access-45nbg\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.406122 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.406004 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3a9bb7-0493-4eef-b193-035ef927d453-openshift-service-ca\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.449572 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.449538 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgx9m\"" Nov 03 13:11:52.449572 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.449548 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 13:11:52.449804 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.449624 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 13:11:52.450802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.450785 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c645947f-qmmh7"] Nov 03 13:11:52.506327 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.506293 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3a9bb7-0493-4eef-b193-035ef927d453-openshift-service-ca\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.506607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.506350 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-metrics-certs\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.506607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.506404 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-maas-api-tls\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.506607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.506426 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45nbg\" (UniqueName: \"kubernetes.io/projected/ed3a9bb7-0493-4eef-b193-035ef927d453-kube-api-access-45nbg\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.507530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.507495 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3a9bb7-0493-4eef-b193-035ef927d453-openshift-service-ca\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.509128 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.509109 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-metrics-certs\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.509382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.509359 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-maas-api-tls\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.560273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.560229 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45nbg\" (UniqueName: \"kubernetes.io/projected/ed3a9bb7-0493-4eef-b193-035ef927d453-kube-api-access-45nbg\") pod \"maas-api-68c645947f-qmmh7\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:52.661864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:52.661832 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:53.021351 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:53.021312 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c645947f-qmmh7"] Nov 03 13:11:53.023970 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:11:53.023943 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3a9bb7_0493_4eef_b193_035ef927d453.slice/crio-6d5ca9cdd01908dd0355967a8b0846dfba28807daff8879b4af14ef294a72495 WatchSource:0}: Error finding container 6d5ca9cdd01908dd0355967a8b0846dfba28807daff8879b4af14ef294a72495: Status 404 returned error can't find the container with id 6d5ca9cdd01908dd0355967a8b0846dfba28807daff8879b4af14ef294a72495 Nov 03 13:11:53.497841 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:53.497806 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" event={"ID":"ed3a9bb7-0493-4eef-b193-035ef927d453","Type":"ContainerStarted","Data":"6d5ca9cdd01908dd0355967a8b0846dfba28807daff8879b4af14ef294a72495"} Nov 03 13:11:59.523524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:59.523429 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" event={"ID":"ed3a9bb7-0493-4eef-b193-035ef927d453","Type":"ContainerStarted","Data":"d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5"} Nov 03 13:11:59.523923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:59.523535 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:11:59.632737 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:11:59.632684 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" podStartSLOduration=1.464726804 podStartE2EDuration="7.632670393s" totalImagesPullingTime="6.167943589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:59.631318139 +0000 UTC m=+450.945289998" watchObservedRunningTime="2026-11-03 13:11:59.632670393 +0000 UTC m=+450.946642253" Nov 03 13:12:05.533576 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:05.533547 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:12:06.366041 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.366006 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lz89b"] Nov 03 13:12:06.368305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.368289 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" Nov 03 13:12:06.411809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.411754 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thmg\" (UniqueName: \"kubernetes.io/projected/33e9a98c-9bdf-4832-8725-2a5ed0b193ec-kube-api-access-8thmg\") pod \"authorino-6f6944bcdd-lz89b\" (UID: \"33e9a98c-9bdf-4832-8725-2a5ed0b193ec\") " pod="kuadrant-system/authorino-6f6944bcdd-lz89b" Nov 03 13:12:06.417730 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.417709 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-swnn6\"" Nov 03 13:12:06.420880 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.420855 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 13:12:06.420986 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.420904 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 13:12:06.466030 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.465996 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lz89b"] Nov 03 13:12:06.512184 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.512141 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thmg\" (UniqueName: \"kubernetes.io/projected/33e9a98c-9bdf-4832-8725-2a5ed0b193ec-kube-api-access-8thmg\") pod \"authorino-6f6944bcdd-lz89b\" (UID: \"33e9a98c-9bdf-4832-8725-2a5ed0b193ec\") " pod="kuadrant-system/authorino-6f6944bcdd-lz89b" Nov 03 13:12:06.543106 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.543066 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm"] Nov 03 13:12:06.545298 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.545282 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.565920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.565892 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thmg\" (UniqueName: \"kubernetes.io/projected/33e9a98c-9bdf-4832-8725-2a5ed0b193ec-kube-api-access-8thmg\") pod \"authorino-6f6944bcdd-lz89b\" (UID: \"33e9a98c-9bdf-4832-8725-2a5ed0b193ec\") " pod="kuadrant-system/authorino-6f6944bcdd-lz89b" Nov 03 13:12:06.600069 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.600030 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm"] Nov 03 13:12:06.613148 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.613122 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-maas-api-tls\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.613288 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.613163 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wg8f\" (UniqueName: \"kubernetes.io/projected/0094c30d-13f8-4f1d-b51a-69260417fe0b-kube-api-access-4wg8f\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.613288 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.613256 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094c30d-13f8-4f1d-b51a-69260417fe0b-openshift-service-ca\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.613378 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.613297 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-metrics-certs\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.688132 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.688097 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" Nov 03 13:12:06.713237 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.713211 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lz89b"] Nov 03 13:12:06.713935 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.713911 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094c30d-13f8-4f1d-b51a-69260417fe0b-openshift-service-ca\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.714070 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.713954 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-metrics-certs\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.714070 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.714008 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-maas-api-tls\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.714070 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.714046 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wg8f\" (UniqueName: \"kubernetes.io/projected/0094c30d-13f8-4f1d-b51a-69260417fe0b-kube-api-access-4wg8f\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.714628 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.714606 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094c30d-13f8-4f1d-b51a-69260417fe0b-openshift-service-ca\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.716576 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.716552 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-maas-api-tls\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.716760 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.716738 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-metrics-certs\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.733577 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.733548 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-dd45b6585-klv5z"] Nov 03 13:12:06.736031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.736011 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd45b6585-klv5z" Nov 03 13:12:06.768541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.768516 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wg8f\" (UniqueName: \"kubernetes.io/projected/0094c30d-13f8-4f1d-b51a-69260417fe0b-kube-api-access-4wg8f\") pod \"maas-api-5b8b4cb57b-bc7hm\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.789892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.789857 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-dd45b6585-klv5z"] Nov 03 13:12:06.815204 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.815173 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mxpx\" (UniqueName: \"kubernetes.io/projected/2e3afef1-3df8-4631-b497-f173d922aacf-kube-api-access-7mxpx\") pod \"authorino-dd45b6585-klv5z\" (UID: \"2e3afef1-3df8-4631-b497-f173d922aacf\") " pod="kuadrant-system/authorino-dd45b6585-klv5z" Nov 03 13:12:06.857207 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.857169 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:06.916279 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.916240 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mxpx\" (UniqueName: \"kubernetes.io/projected/2e3afef1-3df8-4631-b497-f173d922aacf-kube-api-access-7mxpx\") pod \"authorino-dd45b6585-klv5z\" (UID: \"2e3afef1-3df8-4631-b497-f173d922aacf\") " pod="kuadrant-system/authorino-dd45b6585-klv5z" Nov 03 13:12:06.969926 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:06.969857 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mxpx\" (UniqueName: \"kubernetes.io/projected/2e3afef1-3df8-4631-b497-f173d922aacf-kube-api-access-7mxpx\") pod \"authorino-dd45b6585-klv5z\" (UID: \"2e3afef1-3df8-4631-b497-f173d922aacf\") " pod="kuadrant-system/authorino-dd45b6585-klv5z" Nov 03 13:12:07.050261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.050214 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd45b6585-klv5z" Nov 03 13:12:07.053097 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.053072 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lz89b"] Nov 03 13:12:07.055531 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:12:07.055503 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33e9a98c_9bdf_4832_8725_2a5ed0b193ec.slice/crio-ade5247a92015c3e8dece0825f6211d68709bcfadbdfd12ec2500b7a9fa94008 WatchSource:0}: Error finding container ade5247a92015c3e8dece0825f6211d68709bcfadbdfd12ec2500b7a9fa94008: Status 404 returned error can't find the container with id ade5247a92015c3e8dece0825f6211d68709bcfadbdfd12ec2500b7a9fa94008 Nov 03 13:12:07.194724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.194687 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dd45b6585-klv5z"] Nov 03 13:12:07.215489 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.215454 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65bf5b6f59-2k52j"] Nov 03 13:12:07.218722 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.218678 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:12:07.219084 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.219059 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm"] Nov 03 13:12:07.222588 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:12:07.222565 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0094c30d_13f8_4f1d_b51a_69260417fe0b.slice/crio-d7a45f70b270f7743b84eff82b8dde26ba568b0d97014701abab4b6338d197e2 WatchSource:0}: Error finding container d7a45f70b270f7743b84eff82b8dde26ba568b0d97014701abab4b6338d197e2: Status 404 returned error can't find the container with id d7a45f70b270f7743b84eff82b8dde26ba568b0d97014701abab4b6338d197e2 Nov 03 13:12:07.271818 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.271766 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 13:12:07.273285 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.273254 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65bf5b6f59-2k52j"] Nov 03 13:12:07.319178 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.319148 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9sm5\" (UniqueName: \"kubernetes.io/projected/f97b32d7-7d37-4f72-a40e-5aea48789ff5-kube-api-access-z9sm5\") pod \"authorino-65bf5b6f59-2k52j\" (UID: \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\") " pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:12:07.319349 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.319212 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f97b32d7-7d37-4f72-a40e-5aea48789ff5-tls-cert\") pod \"authorino-65bf5b6f59-2k52j\" (UID: \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\") " pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:12:07.407395 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.407371 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dd45b6585-klv5z"] Nov 03 13:12:07.409859 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:12:07.409832 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e3afef1_3df8_4631_b497_f173d922aacf.slice/crio-afc783089daf2651eb78abdf9c510d45ce8bab861e1f8d76273ced6405f9e167 WatchSource:0}: Error finding container afc783089daf2651eb78abdf9c510d45ce8bab861e1f8d76273ced6405f9e167: Status 404 returned error can't find the container with id afc783089daf2651eb78abdf9c510d45ce8bab861e1f8d76273ced6405f9e167 Nov 03 13:12:07.420260 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.420229 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f97b32d7-7d37-4f72-a40e-5aea48789ff5-tls-cert\") pod \"authorino-65bf5b6f59-2k52j\" (UID: \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\") " pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:12:07.420363 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.420294 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9sm5\" (UniqueName: \"kubernetes.io/projected/f97b32d7-7d37-4f72-a40e-5aea48789ff5-kube-api-access-z9sm5\") pod \"authorino-65bf5b6f59-2k52j\" (UID: \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\") " pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:12:07.422556 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.422534 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f97b32d7-7d37-4f72-a40e-5aea48789ff5-tls-cert\") pod \"authorino-65bf5b6f59-2k52j\" (UID: \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\") " pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:12:07.473806 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.473706 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9sm5\" (UniqueName: \"kubernetes.io/projected/f97b32d7-7d37-4f72-a40e-5aea48789ff5-kube-api-access-z9sm5\") pod \"authorino-65bf5b6f59-2k52j\" (UID: \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\") " pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:12:07.529493 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.529464 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:12:07.560518 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.560480 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd45b6585-klv5z" event={"ID":"2e3afef1-3df8-4631-b497-f173d922aacf","Type":"ContainerStarted","Data":"afc783089daf2651eb78abdf9c510d45ce8bab861e1f8d76273ced6405f9e167"} Nov 03 13:12:07.561712 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.561674 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" event={"ID":"33e9a98c-9bdf-4832-8725-2a5ed0b193ec","Type":"ContainerStarted","Data":"ade5247a92015c3e8dece0825f6211d68709bcfadbdfd12ec2500b7a9fa94008"} Nov 03 13:12:07.562891 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.562853 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" event={"ID":"0094c30d-13f8-4f1d-b51a-69260417fe0b","Type":"ContainerStarted","Data":"d7a45f70b270f7743b84eff82b8dde26ba568b0d97014701abab4b6338d197e2"} Nov 03 13:12:07.897566 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:07.893916 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65bf5b6f59-2k52j"] Nov 03 13:12:07.899327 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:12:07.899298 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf97b32d7_7d37_4f72_a40e_5aea48789ff5.slice/crio-0e06e970bff2b18a46d441cd55364a6337bf97d534e465e187ff67a2b67d7561 WatchSource:0}: Error finding container 0e06e970bff2b18a46d441cd55364a6337bf97d534e465e187ff67a2b67d7561: Status 404 returned error can't find the container with id 0e06e970bff2b18a46d441cd55364a6337bf97d534e465e187ff67a2b67d7561 Nov 03 13:12:08.567920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:08.567875 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" event={"ID":"0094c30d-13f8-4f1d-b51a-69260417fe0b","Type":"ContainerStarted","Data":"7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f"} Nov 03 13:12:08.568386 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:08.567982 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:08.569098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:08.569077 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" event={"ID":"f97b32d7-7d37-4f72-a40e-5aea48789ff5","Type":"ContainerStarted","Data":"0e06e970bff2b18a46d441cd55364a6337bf97d534e465e187ff67a2b67d7561"} Nov 03 13:12:08.675745 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:08.675684 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" podStartSLOduration=1.841276185 podStartE2EDuration="2.675669922s" totalImagesPullingTime="834.393737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:12:08.675193177 +0000 UTC m=+459.989165036" watchObservedRunningTime="2026-11-03 13:12:08.675669922 +0000 UTC m=+459.989641782" Nov 03 13:12:12.590682 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.590644 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd45b6585-klv5z" event={"ID":"2e3afef1-3df8-4631-b497-f173d922aacf","Type":"ContainerStarted","Data":"c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50"} Nov 03 13:12:12.591175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.590728 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-dd45b6585-klv5z" podUID="2e3afef1-3df8-4631-b497-f173d922aacf" containerName="authorino" containerID="cri-o://c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50" gracePeriod=30 Nov 03 13:12:12.592216 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.592185 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" event={"ID":"33e9a98c-9bdf-4832-8725-2a5ed0b193ec","Type":"ContainerStarted","Data":"d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e"} Nov 03 13:12:12.592334 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.592226 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" podUID="33e9a98c-9bdf-4832-8725-2a5ed0b193ec" containerName="authorino" containerID="cri-o://d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e" gracePeriod=30 Nov 03 13:12:12.593728 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.593703 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" event={"ID":"f97b32d7-7d37-4f72-a40e-5aea48789ff5","Type":"ContainerStarted","Data":"6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec"} Nov 03 13:12:12.698383 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.698323 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-dd45b6585-klv5z" podStartSLOduration=2.113015921 podStartE2EDuration="6.698307776s" totalImagesPullingTime="4.585291855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:12:12.697090332 +0000 UTC m=+464.011062194" watchObservedRunningTime="2026-11-03 13:12:12.698307776 +0000 UTC m=+464.012279636" Nov 03 13:12:12.806663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.806596 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" podStartSLOduration=1.543438875 podStartE2EDuration="5.806577489s" totalImagesPullingTime="4.263138614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:12:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:12:12.804094834 +0000 UTC m=+464.118066695" watchObservedRunningTime="2026-11-03 13:12:12.806577489 +0000 UTC m=+464.120549349" Nov 03 13:12:12.873488 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.873458 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" Nov 03 13:12:12.875412 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.875385 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd45b6585-klv5z" Nov 03 13:12:12.910466 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:12.910412 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" podStartSLOduration=1.8531976540000001 podStartE2EDuration="6.910398073s" totalImagesPullingTime="5.057200419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:12:12.910056689 +0000 UTC m=+464.224028548" watchObservedRunningTime="2026-11-03 13:12:12.910398073 +0000 UTC m=+464.224369927" Nov 03 13:12:13.000328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.000291 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3afef1-3df8-4631-b497-f173d922aacf-kube-api-access-7mxpx\" (UniqueName: \"kubernetes.io/projected/2e3afef1-3df8-4631-b497-f173d922aacf-kube-api-access-7mxpx\") pod \"2e3afef1-3df8-4631-b497-f173d922aacf\" (UID: \"2e3afef1-3df8-4631-b497-f173d922aacf\") " Nov 03 13:12:13.000517 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.000347 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e9a98c-9bdf-4832-8725-2a5ed0b193ec-kube-api-access-8thmg\" (UniqueName: \"kubernetes.io/projected/33e9a98c-9bdf-4832-8725-2a5ed0b193ec-kube-api-access-8thmg\") pod \"33e9a98c-9bdf-4832-8725-2a5ed0b193ec\" (UID: \"33e9a98c-9bdf-4832-8725-2a5ed0b193ec\") " Nov 03 13:12:13.002544 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.002512 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e9a98c-9bdf-4832-8725-2a5ed0b193ec-kube-api-access-8thmg" pod "33e9a98c-9bdf-4832-8725-2a5ed0b193ec" (UID: "33e9a98c-9bdf-4832-8725-2a5ed0b193ec"). InnerVolumeSpecName "kube-api-access-8thmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:12:13.002657 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.002565 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3afef1-3df8-4631-b497-f173d922aacf-kube-api-access-7mxpx" pod "2e3afef1-3df8-4631-b497-f173d922aacf" (UID: "2e3afef1-3df8-4631-b497-f173d922aacf"). InnerVolumeSpecName "kube-api-access-7mxpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:12:13.101587 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.101545 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mxpx\" (UniqueName: \"kubernetes.io/projected/2e3afef1-3df8-4631-b497-f173d922aacf-kube-api-access-7mxpx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:12:13.101587 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.101577 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8thmg\" (UniqueName: \"kubernetes.io/projected/33e9a98c-9bdf-4832-8725-2a5ed0b193ec-kube-api-access-8thmg\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:12:13.598891 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.598860 4541 generic.go:356] "Generic (PLEG): container finished" podID="2e3afef1-3df8-4631-b497-f173d922aacf" containerID="c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50" exitCode=0 Nov 03 13:12:13.599355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.598902 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-dd45b6585-klv5z" Nov 03 13:12:13.599355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.598935 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd45b6585-klv5z" event={"ID":"2e3afef1-3df8-4631-b497-f173d922aacf","Type":"ContainerDied","Data":"c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50"} Nov 03 13:12:13.599355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.598962 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-dd45b6585-klv5z" event={"ID":"2e3afef1-3df8-4631-b497-f173d922aacf","Type":"ContainerDied","Data":"afc783089daf2651eb78abdf9c510d45ce8bab861e1f8d76273ced6405f9e167"} Nov 03 13:12:13.599355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.598978 4541 scope.go:122] "RemoveContainer" containerID="c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50" Nov 03 13:12:13.600387 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.600367 4541 generic.go:356] "Generic (PLEG): container finished" podID="33e9a98c-9bdf-4832-8725-2a5ed0b193ec" containerID="d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e" exitCode=0 Nov 03 13:12:13.600497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.600406 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" Nov 03 13:12:13.600497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.600432 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" event={"ID":"33e9a98c-9bdf-4832-8725-2a5ed0b193ec","Type":"ContainerDied","Data":"d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e"} Nov 03 13:12:13.600497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.600473 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-lz89b" event={"ID":"33e9a98c-9bdf-4832-8725-2a5ed0b193ec","Type":"ContainerDied","Data":"ade5247a92015c3e8dece0825f6211d68709bcfadbdfd12ec2500b7a9fa94008"} Nov 03 13:12:13.608148 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.608128 4541 scope.go:122] "RemoveContainer" containerID="c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50" Nov 03 13:12:13.608669 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:12:13.608404 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50\": container with ID starting with c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50 not found: ID does not exist" containerID="c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50" Nov 03 13:12:13.608669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.608432 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50"} err="failed to get container status \"c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50\": rpc error: code = NotFound desc = could not find container \"c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50\": container with ID starting with c9191fe6bd307ef618eda466bb94536875e8fff99950fc57391a7e3c7be9ed50 not found: ID does not exist" Nov 03 13:12:13.608669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.608451 4541 scope.go:122] "RemoveContainer" containerID="d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e" Nov 03 13:12:13.616087 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.616069 4541 scope.go:122] "RemoveContainer" containerID="d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e" Nov 03 13:12:13.616366 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:12:13.616349 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e\": container with ID starting with d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e not found: ID does not exist" containerID="d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e" Nov 03 13:12:13.616404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.616373 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e"} err="failed to get container status \"d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e\": rpc error: code = NotFound desc = could not find container \"d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e\": container with ID starting with d66a381ea1775a37d8ec32b9151a5609dc17d439032926486ba40a79347bdb8e not found: ID does not exist" Nov 03 13:12:13.751247 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.751211 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lz89b"] Nov 03 13:12:13.798002 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.797967 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-lz89b"] Nov 03 13:12:13.900461 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.900427 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-dd45b6585-klv5z"] Nov 03 13:12:13.904390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:13.904364 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-dd45b6585-klv5z"] Nov 03 13:12:14.581716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:14.581688 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 13:12:14.739007 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:14.738976 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c645947f-qmmh7"] Nov 03 13:12:14.739480 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:14.739166 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" podUID="ed3a9bb7-0493-4eef-b193-035ef927d453" containerName="maas-api" containerID="cri-o://d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5" gracePeriod=30 Nov 03 13:12:14.866192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:14.866164 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3afef1-3df8-4631-b497-f173d922aacf" path="/var/lib/kubelet/pods/2e3afef1-3df8-4631-b497-f173d922aacf/volumes" Nov 03 13:12:14.866515 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:14.866501 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e9a98c-9bdf-4832-8725-2a5ed0b193ec" path="/var/lib/kubelet/pods/33e9a98c-9bdf-4832-8725-2a5ed0b193ec/volumes" Nov 03 13:12:14.978384 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:14.978360 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:12:15.119703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.119625 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3a9bb7-0493-4eef-b193-035ef927d453-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3a9bb7-0493-4eef-b193-035ef927d453-openshift-service-ca\") pod \"ed3a9bb7-0493-4eef-b193-035ef927d453\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " Nov 03 13:12:15.119860 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.119718 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-metrics-certs\") pod \"ed3a9bb7-0493-4eef-b193-035ef927d453\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " Nov 03 13:12:15.119860 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.119737 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-maas-api-tls\") pod \"ed3a9bb7-0493-4eef-b193-035ef927d453\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " Nov 03 13:12:15.119860 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.119760 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3a9bb7-0493-4eef-b193-035ef927d453-kube-api-access-45nbg\" (UniqueName: \"kubernetes.io/projected/ed3a9bb7-0493-4eef-b193-035ef927d453-kube-api-access-45nbg\") pod \"ed3a9bb7-0493-4eef-b193-035ef927d453\" (UID: \"ed3a9bb7-0493-4eef-b193-035ef927d453\") " Nov 03 13:12:15.120096 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.120069 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3a9bb7-0493-4eef-b193-035ef927d453-openshift-service-ca" pod "ed3a9bb7-0493-4eef-b193-035ef927d453" (UID: "ed3a9bb7-0493-4eef-b193-035ef927d453"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:12:15.122036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.122004 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-metrics-certs" pod "ed3a9bb7-0493-4eef-b193-035ef927d453" (UID: "ed3a9bb7-0493-4eef-b193-035ef927d453"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:12:15.122166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.122037 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3a9bb7-0493-4eef-b193-035ef927d453-kube-api-access-45nbg" pod "ed3a9bb7-0493-4eef-b193-035ef927d453" (UID: "ed3a9bb7-0493-4eef-b193-035ef927d453"). InnerVolumeSpecName "kube-api-access-45nbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:12:15.122166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.122113 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-maas-api-tls" pod "ed3a9bb7-0493-4eef-b193-035ef927d453" (UID: "ed3a9bb7-0493-4eef-b193-035ef927d453"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:12:15.221511 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.221469 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:12:15.221511 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.221509 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed3a9bb7-0493-4eef-b193-035ef927d453-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:12:15.221708 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.221526 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45nbg\" (UniqueName: \"kubernetes.io/projected/ed3a9bb7-0493-4eef-b193-035ef927d453-kube-api-access-45nbg\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:12:15.221708 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.221558 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3a9bb7-0493-4eef-b193-035ef927d453-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:12:15.611166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.611130 4541 generic.go:356] "Generic (PLEG): container finished" podID="ed3a9bb7-0493-4eef-b193-035ef927d453" containerID="d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5" exitCode=0 Nov 03 13:12:15.611329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.611178 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" Nov 03 13:12:15.611329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.611220 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" event={"ID":"ed3a9bb7-0493-4eef-b193-035ef927d453","Type":"ContainerDied","Data":"d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5"} Nov 03 13:12:15.611329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.611270 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c645947f-qmmh7" event={"ID":"ed3a9bb7-0493-4eef-b193-035ef927d453","Type":"ContainerDied","Data":"6d5ca9cdd01908dd0355967a8b0846dfba28807daff8879b4af14ef294a72495"} Nov 03 13:12:15.611329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.611295 4541 scope.go:122] "RemoveContainer" containerID="d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5" Nov 03 13:12:15.620447 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.620429 4541 scope.go:122] "RemoveContainer" containerID="d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5" Nov 03 13:12:15.620696 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:12:15.620673 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5\": container with ID starting with d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5 not found: ID does not exist" containerID="d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5" Nov 03 13:12:15.620765 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.620702 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5"} err="failed to get container status \"d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5\": rpc error: code = NotFound desc = could not find container \"d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5\": container with ID starting with d72c811a6e82e048700a61f7d3b15efbdaca0adda9224994f355424e32acd6c5 not found: ID does not exist" Nov 03 13:12:15.764426 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.764393 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c645947f-qmmh7"] Nov 03 13:12:15.811716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:15.811678 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c645947f-qmmh7"] Nov 03 13:12:16.866394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:12:16.866361 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3a9bb7-0493-4eef-b193-035ef927d453" path="/var/lib/kubelet/pods/ed3a9bb7-0493-4eef-b193-035ef927d453/volumes" Nov 03 13:12:16.888515 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:12:16.888483 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:12:46.889181 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:12:46.889138 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:13:16.887766 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:13:16.887725 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:13:28.955230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955194 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-nb4bb"] Nov 03 13:13:28.955654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955461 4541 cpu_manager.go:414] "containerMap: removing container" podUID="33e9a98c-9bdf-4832-8725-2a5ed0b193ec" containerName="authorino" Nov 03 13:13:28.955654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955471 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e9a98c-9bdf-4832-8725-2a5ed0b193ec" containerName="authorino" Nov 03 13:13:28.955654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955496 4541 cpu_manager.go:414] "containerMap: removing container" podUID="2e3afef1-3df8-4631-b497-f173d922aacf" containerName="authorino" Nov 03 13:13:28.955654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955501 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3afef1-3df8-4631-b497-f173d922aacf" containerName="authorino" Nov 03 13:13:28.955654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955509 4541 cpu_manager.go:414] "containerMap: removing container" podUID="ed3a9bb7-0493-4eef-b193-035ef927d453" containerName="maas-api" Nov 03 13:13:28.955654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955515 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3a9bb7-0493-4eef-b193-035ef927d453" containerName="maas-api" Nov 03 13:13:28.955654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955557 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3afef1-3df8-4631-b497-f173d922aacf" containerName="authorino" Nov 03 13:13:28.955654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955565 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3a9bb7-0493-4eef-b193-035ef927d453" containerName="maas-api" Nov 03 13:13:28.955654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.955570 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e9a98c-9bdf-4832-8725-2a5ed0b193ec" containerName="authorino" Nov 03 13:13:28.957433 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:28.957417 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:29.054060 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.054022 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 13:13:29.054261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.054084 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 13:13:29.054261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.054087 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 13:13:29.054261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.054137 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xqvqk\"" Nov 03 13:13:29.054261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.054170 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 13:13:29.055288 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.055269 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-nb4bb"] Nov 03 13:13:29.065610 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.065574 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbfr8\" (UniqueName: \"kubernetes.io/projected/6b3010a4-fb91-480a-bbca-c4471865d292-kube-api-access-dbfr8\") pod \"ai-gateway-controller-5d8dddb676-nb4bb\" (UID: \"6b3010a4-fb91-480a-bbca-c4471865d292\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:29.065909 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.065882 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3010a4-fb91-480a-bbca-c4471865d292-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-nb4bb\" (UID: \"6b3010a4-fb91-480a-bbca-c4471865d292\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:29.166930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.166886 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3010a4-fb91-480a-bbca-c4471865d292-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-nb4bb\" (UID: \"6b3010a4-fb91-480a-bbca-c4471865d292\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:29.167108 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.166949 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbfr8\" (UniqueName: \"kubernetes.io/projected/6b3010a4-fb91-480a-bbca-c4471865d292-kube-api-access-dbfr8\") pod \"ai-gateway-controller-5d8dddb676-nb4bb\" (UID: \"6b3010a4-fb91-480a-bbca-c4471865d292\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:29.169599 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.169564 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3010a4-fb91-480a-bbca-c4471865d292-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-nb4bb\" (UID: \"6b3010a4-fb91-480a-bbca-c4471865d292\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:29.220723 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.220642 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbfr8\" (UniqueName: \"kubernetes.io/projected/6b3010a4-fb91-480a-bbca-c4471865d292-kube-api-access-dbfr8\") pod \"ai-gateway-controller-5d8dddb676-nb4bb\" (UID: \"6b3010a4-fb91-480a-bbca-c4471865d292\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:29.275901 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.275867 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:29.635811 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.635762 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-nb4bb"] Nov 03 13:13:29.638254 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:13:29.638215 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3010a4_fb91_480a_bbca_c4471865d292.slice/crio-197bfc9c781f5f1f7e0be356c3e4b9fd27b8c981481c37286f2011ffe4d9e777 WatchSource:0}: Error finding container 197bfc9c781f5f1f7e0be356c3e4b9fd27b8c981481c37286f2011ffe4d9e777: Status 404 returned error can't find the container with id 197bfc9c781f5f1f7e0be356c3e4b9fd27b8c981481c37286f2011ffe4d9e777 Nov 03 13:13:29.883779 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:29.883743 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" event={"ID":"6b3010a4-fb91-480a-bbca-c4471865d292","Type":"ContainerStarted","Data":"197bfc9c781f5f1f7e0be356c3e4b9fd27b8c981481c37286f2011ffe4d9e777"} Nov 03 13:13:31.893231 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:31.893142 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" event={"ID":"6b3010a4-fb91-480a-bbca-c4471865d292","Type":"ContainerStarted","Data":"6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae"} Nov 03 13:13:31.893603 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:31.893266 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:35.781053 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.780996 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" podStartSLOduration=5.8552884370000005 podStartE2EDuration="7.780980835s" totalImagesPullingTime="1.925692398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:32.05386178 +0000 UTC m=+543.367833640" watchObservedRunningTime="2026-11-03 13:13:35.780980835 +0000 UTC m=+547.094952694" Nov 03 13:13:35.781522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.781506 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h85vt"] Nov 03 13:13:35.783584 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.783569 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:35.833553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.833520 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 13:13:35.880738 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.880704 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lpgh7\"" Nov 03 13:13:35.881898 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.881877 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h85vt"] Nov 03 13:13:35.921452 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.921418 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-h85vt\" (UID: \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:35.921604 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.921527 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r42b\" (UniqueName: \"kubernetes.io/projected/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-kube-api-access-5r42b\") pod \"payload-pre-processing-7b98bb678c-h85vt\" (UID: \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:35.936139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.936103 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qnx22"] Nov 03 13:13:35.938058 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:35.938041 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:36.001851 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.001812 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qnx22"] Nov 03 13:13:36.022398 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.022365 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r42b\" (UniqueName: \"kubernetes.io/projected/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-kube-api-access-5r42b\") pod \"payload-pre-processing-7b98bb678c-h85vt\" (UID: \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:36.022567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.022409 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6b3624-855b-4ac2-88d4-cfa28b23a721-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-qnx22\" (UID: \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:36.022567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.022441 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-h85vt\" (UID: \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:36.022567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.022471 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tgq6\" (UniqueName: \"kubernetes.io/projected/bf6b3624-855b-4ac2-88d4-cfa28b23a721-kube-api-access-5tgq6\") pod \"payload-processing-55fbbbf9dc-qnx22\" (UID: \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:36.023227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.023202 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-h85vt\" (UID: \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:36.076865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.076787 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r42b\" (UniqueName: \"kubernetes.io/projected/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-kube-api-access-5r42b\") pod \"payload-pre-processing-7b98bb678c-h85vt\" (UID: \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:36.094401 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.094369 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:36.123532 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.123495 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tgq6\" (UniqueName: \"kubernetes.io/projected/bf6b3624-855b-4ac2-88d4-cfa28b23a721-kube-api-access-5tgq6\") pod \"payload-processing-55fbbbf9dc-qnx22\" (UID: \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:36.123710 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.123584 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6b3624-855b-4ac2-88d4-cfa28b23a721-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-qnx22\" (UID: \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:36.124300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.124275 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6b3624-855b-4ac2-88d4-cfa28b23a721-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-qnx22\" (UID: \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:36.178275 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.178237 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tgq6\" (UniqueName: \"kubernetes.io/projected/bf6b3624-855b-4ac2-88d4-cfa28b23a721-kube-api-access-5tgq6\") pod \"payload-processing-55fbbbf9dc-qnx22\" (UID: \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:36.249328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.249284 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:36.454567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.454543 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h85vt"] Nov 03 13:13:36.456476 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:13:36.456451 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392c3cf5_f8f8_4cdb_ba5c_23c0230caf27.slice/crio-76dfbd1394c4b64caaf0df4eab771270d2c9f2ea676053d7c1dcf47ae4659dc9 WatchSource:0}: Error finding container 76dfbd1394c4b64caaf0df4eab771270d2c9f2ea676053d7c1dcf47ae4659dc9: Status 404 returned error can't find the container with id 76dfbd1394c4b64caaf0df4eab771270d2c9f2ea676053d7c1dcf47ae4659dc9 Nov 03 13:13:36.611230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.611183 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qnx22"] Nov 03 13:13:36.613349 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:13:36.613315 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6b3624_855b_4ac2_88d4_cfa28b23a721.slice/crio-473c0a49c9291b5c395676416c7047af6879158a94ef7c5f1112b10e60d2ad99 WatchSource:0}: Error finding container 473c0a49c9291b5c395676416c7047af6879158a94ef7c5f1112b10e60d2ad99: Status 404 returned error can't find the container with id 473c0a49c9291b5c395676416c7047af6879158a94ef7c5f1112b10e60d2ad99 Nov 03 13:13:36.911896 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.911856 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" event={"ID":"bf6b3624-855b-4ac2-88d4-cfa28b23a721","Type":"ContainerStarted","Data":"473c0a49c9291b5c395676416c7047af6879158a94ef7c5f1112b10e60d2ad99"} Nov 03 13:13:36.913036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:36.913015 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" event={"ID":"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27","Type":"ContainerStarted","Data":"76dfbd1394c4b64caaf0df4eab771270d2c9f2ea676053d7c1dcf47ae4659dc9"} Nov 03 13:13:39.928267 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:39.928221 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" event={"ID":"bf6b3624-855b-4ac2-88d4-cfa28b23a721","Type":"ContainerStarted","Data":"2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044"} Nov 03 13:13:39.928725 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:39.928294 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:39.929720 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:39.929691 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" event={"ID":"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27","Type":"ContainerStarted","Data":"623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577"} Nov 03 13:13:39.929868 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:39.929814 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:40.040276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:40.040215 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" podStartSLOduration=2.131926327 podStartE2EDuration="5.040200985s" totalImagesPullingTime="2.908274658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:40.038912294 +0000 UTC m=+551.352884157" watchObservedRunningTime="2026-11-03 13:13:40.040200985 +0000 UTC m=+551.354172844" Nov 03 13:13:40.147889 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:40.147840 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" podStartSLOduration=2.120351251 podStartE2EDuration="5.14782612s" totalImagesPullingTime="3.027474869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:40.146719478 +0000 UTC m=+551.460691337" watchObservedRunningTime="2026-11-03 13:13:40.14782612 +0000 UTC m=+551.461797979" Nov 03 13:13:42.899078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:42.899047 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:43.250139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.250049 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-cb4ffdf47-czmlj"] Nov 03 13:13:43.253309 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.253282 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:43.307944 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.307904 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-cb4ffdf47-czmlj"] Nov 03 13:13:43.385398 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.385362 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb24765f-1472-41e4-863d-ecdcc9f1ee97-metrics-certs\") pod \"ai-gateway-controller-cb4ffdf47-czmlj\" (UID: \"bb24765f-1472-41e4-863d-ecdcc9f1ee97\") " pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:43.385569 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.385414 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lf6b\" (UniqueName: \"kubernetes.io/projected/bb24765f-1472-41e4-863d-ecdcc9f1ee97-kube-api-access-2lf6b\") pod \"ai-gateway-controller-cb4ffdf47-czmlj\" (UID: \"bb24765f-1472-41e4-863d-ecdcc9f1ee97\") " pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:43.486537 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.486501 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb24765f-1472-41e4-863d-ecdcc9f1ee97-metrics-certs\") pod \"ai-gateway-controller-cb4ffdf47-czmlj\" (UID: \"bb24765f-1472-41e4-863d-ecdcc9f1ee97\") " pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:43.486707 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.486554 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lf6b\" (UniqueName: \"kubernetes.io/projected/bb24765f-1472-41e4-863d-ecdcc9f1ee97-kube-api-access-2lf6b\") pod \"ai-gateway-controller-cb4ffdf47-czmlj\" (UID: \"bb24765f-1472-41e4-863d-ecdcc9f1ee97\") " pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:43.488952 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.488929 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb24765f-1472-41e4-863d-ecdcc9f1ee97-metrics-certs\") pod \"ai-gateway-controller-cb4ffdf47-czmlj\" (UID: \"bb24765f-1472-41e4-863d-ecdcc9f1ee97\") " pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:43.541137 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.541060 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lf6b\" (UniqueName: \"kubernetes.io/projected/bb24765f-1472-41e4-863d-ecdcc9f1ee97-kube-api-access-2lf6b\") pod \"ai-gateway-controller-cb4ffdf47-czmlj\" (UID: \"bb24765f-1472-41e4-863d-ecdcc9f1ee97\") " pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:43.565741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.565699 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:43.926248 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.926219 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-cb4ffdf47-czmlj"] Nov 03 13:13:43.928753 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:13:43.928717 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb24765f_1472_41e4_863d_ecdcc9f1ee97.slice/crio-55891c590f2d379130c712ba7ae7aec9ec298a74b3e0f3443aa656cecfd2d05c WatchSource:0}: Error finding container 55891c590f2d379130c712ba7ae7aec9ec298a74b3e0f3443aa656cecfd2d05c: Status 404 returned error can't find the container with id 55891c590f2d379130c712ba7ae7aec9ec298a74b3e0f3443aa656cecfd2d05c Nov 03 13:13:43.948222 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:43.948193 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" event={"ID":"bb24765f-1472-41e4-863d-ecdcc9f1ee97","Type":"ContainerStarted","Data":"55891c590f2d379130c712ba7ae7aec9ec298a74b3e0f3443aa656cecfd2d05c"} Nov 03 13:13:44.953516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:44.953485 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" event={"ID":"bb24765f-1472-41e4-863d-ecdcc9f1ee97","Type":"ContainerStarted","Data":"5a50d30f73bb5a15defa2e61ba84870e33d87151fa69472742388e4b611749b1"} Nov 03 13:13:44.953900 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:44.953626 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:45.064561 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:45.064455 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" podStartSLOduration=1.207912655 podStartE2EDuration="2.064440059s" totalImagesPullingTime="856.527404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:45.063221762 +0000 UTC m=+556.377193625" watchObservedRunningTime="2026-11-03 13:13:45.064440059 +0000 UTC m=+556.378411919" Nov 03 13:13:46.888387 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:13:46.888354 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:13:50.937147 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:50.937117 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 13:13:50.941589 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:50.937198 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 13:13:55.959898 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:55.959865 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-cb4ffdf47-czmlj" Nov 03 13:13:56.118221 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:56.118188 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-nb4bb"] Nov 03 13:13:56.118485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:56.118440 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" podUID="6b3010a4-fb91-480a-bbca-c4471865d292" containerName="manager" containerID="cri-o://6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae" gracePeriod=30 Nov 03 13:13:56.352019 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:56.351995 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:56.492736 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:56.492648 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3010a4-fb91-480a-bbca-c4471865d292-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3010a4-fb91-480a-bbca-c4471865d292-metrics-certs\") pod \"6b3010a4-fb91-480a-bbca-c4471865d292\" (UID: \"6b3010a4-fb91-480a-bbca-c4471865d292\") " Nov 03 13:13:56.492930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:56.492747 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3010a4-fb91-480a-bbca-c4471865d292-kube-api-access-dbfr8\" (UniqueName: \"kubernetes.io/projected/6b3010a4-fb91-480a-bbca-c4471865d292-kube-api-access-dbfr8\") pod \"6b3010a4-fb91-480a-bbca-c4471865d292\" (UID: \"6b3010a4-fb91-480a-bbca-c4471865d292\") " Nov 03 13:13:56.494990 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:56.494954 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3010a4-fb91-480a-bbca-c4471865d292-metrics-certs" pod "6b3010a4-fb91-480a-bbca-c4471865d292" (UID: "6b3010a4-fb91-480a-bbca-c4471865d292"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:13:56.495089 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:56.495066 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3010a4-fb91-480a-bbca-c4471865d292-kube-api-access-dbfr8" pod "6b3010a4-fb91-480a-bbca-c4471865d292" (UID: "6b3010a4-fb91-480a-bbca-c4471865d292"). InnerVolumeSpecName "kube-api-access-dbfr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:13:56.593634 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:56.593590 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbfr8\" (UniqueName: \"kubernetes.io/projected/6b3010a4-fb91-480a-bbca-c4471865d292-kube-api-access-dbfr8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:13:56.593634 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:56.593631 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3010a4-fb91-480a-bbca-c4471865d292-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:13:57.001840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:57.001806 4541 generic.go:356] "Generic (PLEG): container finished" podID="6b3010a4-fb91-480a-bbca-c4471865d292" containerID="6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae" exitCode=0 Nov 03 13:13:57.002210 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:57.001862 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" Nov 03 13:13:57.002210 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:57.001886 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" event={"ID":"6b3010a4-fb91-480a-bbca-c4471865d292","Type":"ContainerDied","Data":"6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae"} Nov 03 13:13:57.002210 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:57.001937 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-nb4bb" event={"ID":"6b3010a4-fb91-480a-bbca-c4471865d292","Type":"ContainerDied","Data":"197bfc9c781f5f1f7e0be356c3e4b9fd27b8c981481c37286f2011ffe4d9e777"} Nov 03 13:13:57.002210 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:57.001960 4541 scope.go:122] "RemoveContainer" containerID="6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae" Nov 03 13:13:57.011744 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:57.011717 4541 scope.go:122] "RemoveContainer" containerID="6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae" Nov 03 13:13:57.012231 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:13:57.012210 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae\": container with ID starting with 6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae not found: ID does not exist" containerID="6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae" Nov 03 13:13:57.012319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:57.012241 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae"} err="failed to get container status \"6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae\": rpc error: code = NotFound desc = could not find container \"6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae\": container with ID starting with 6f7da0ae2615bd0f6b91e45e8927f8c33544cdb18eaaee3d58d06857b57411ae not found: ID does not exist" Nov 03 13:13:57.159050 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:57.159016 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-nb4bb"] Nov 03 13:13:57.206140 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:57.206103 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-nb4bb"] Nov 03 13:13:58.865978 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:13:58.865943 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3010a4-fb91-480a-bbca-c4471865d292" path="/var/lib/kubelet/pods/6b3010a4-fb91-480a-bbca-c4471865d292/volumes" Nov 03 13:14:07.778817 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.778760 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8"] Nov 03 13:14:07.779276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.779117 4541 cpu_manager.go:414] "containerMap: removing container" podUID="6b3010a4-fb91-480a-bbca-c4471865d292" containerName="manager" Nov 03 13:14:07.779276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.779129 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3010a4-fb91-480a-bbca-c4471865d292" containerName="manager" Nov 03 13:14:07.779276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.779187 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3010a4-fb91-480a-bbca-c4471865d292" containerName="manager" Nov 03 13:14:07.782188 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.782169 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.878391 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.878356 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.878555 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.878400 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.878555 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.878423 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4r64\" (UniqueName: \"kubernetes.io/projected/ccd2faed-c412-486e-ba4e-8d103fc677e4-kube-api-access-b4r64\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.878555 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.878489 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.878555 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.878522 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.878555 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.878545 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd2faed-c412-486e-ba4e-8d103fc677e4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.879336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.879311 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6xb8m\"" Nov 03 13:14:07.879762 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.879745 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 13:14:07.879868 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.879803 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 13:14:07.879868 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.879821 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 13:14:07.881039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.881020 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8"] Nov 03 13:14:07.978965 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.978931 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.979157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.978973 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4r64\" (UniqueName: \"kubernetes.io/projected/ccd2faed-c412-486e-ba4e-8d103fc677e4-kube-api-access-b4r64\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.979157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.979005 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.979157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.979038 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.979157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.979076 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd2faed-c412-486e-ba4e-8d103fc677e4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.979157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.979133 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.979407 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.979361 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.979468 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.979432 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.979548 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.979521 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.981462 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.981444 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccd2faed-c412-486e-ba4e-8d103fc677e4-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:07.981566 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:07.981549 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccd2faed-c412-486e-ba4e-8d103fc677e4-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:08.034175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:08.034102 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4r64\" (UniqueName: \"kubernetes.io/projected/ccd2faed-c412-486e-ba4e-8d103fc677e4-kube-api-access-b4r64\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8\" (UID: \"ccd2faed-c412-486e-ba4e-8d103fc677e4\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:08.093904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:08.093863 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:08.461963 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:08.461939 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8"] Nov 03 13:14:08.464466 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:14:08.464431 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccd2faed_c412_486e_ba4e_8d103fc677e4.slice/crio-3e0cab38d086084026925682ed661ef7791913a1c79dadde8971f15a1bd1d112 WatchSource:0}: Error finding container 3e0cab38d086084026925682ed661ef7791913a1c79dadde8971f15a1bd1d112: Status 404 returned error can't find the container with id 3e0cab38d086084026925682ed661ef7791913a1c79dadde8971f15a1bd1d112 Nov 03 13:14:09.055330 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:09.055294 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" event={"ID":"ccd2faed-c412-486e-ba4e-8d103fc677e4","Type":"ContainerStarted","Data":"3e0cab38d086084026925682ed661ef7791913a1c79dadde8971f15a1bd1d112"} Nov 03 13:14:11.065092 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:11.065056 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" event={"ID":"ccd2faed-c412-486e-ba4e-8d103fc677e4","Type":"ContainerStarted","Data":"99ada8b4f186cfdc0ba015d838b4eb87e912176a7e9c97fe2039d022fb155f88"} Nov 03 13:14:11.094443 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:11.094413 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:11.182791 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:11.182719 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" podStartSLOduration=2.192760758 podStartE2EDuration="4.182705888s" totalImagesPullingTime="1.98994513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:11.181688797 +0000 UTC m=+582.495660658" watchObservedRunningTime="2026-11-03 13:14:11.182705888 +0000 UTC m=+582.496677749" Nov 03 13:14:15.444409 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.444372 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79"] Nov 03 13:14:15.446531 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.446515 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.532972 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.532929 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4c2084-9be1-422b-a365-fe3c08cd968c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.533155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.533000 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltnz8\" (UniqueName: \"kubernetes.io/projected/9f4c2084-9be1-422b-a365-fe3c08cd968c-kube-api-access-ltnz8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.533155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.533032 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.533155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.533139 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.533322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.533179 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.533322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.533240 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.542843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.542819 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 03 13:14:15.544150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.544132 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79"] Nov 03 13:14:15.634182 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.634143 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4c2084-9be1-422b-a365-fe3c08cd968c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.634352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.634192 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltnz8\" (UniqueName: \"kubernetes.io/projected/9f4c2084-9be1-422b-a365-fe3c08cd968c-kube-api-access-ltnz8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.634352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.634224 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.634352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.634271 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.634352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.634296 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.634527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.634427 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.634700 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.634682 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.634837 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.634748 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.634894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.634833 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.636510 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.636481 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4c2084-9be1-422b-a365-fe3c08cd968c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.636766 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.636750 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4c2084-9be1-422b-a365-fe3c08cd968c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.688610 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.688570 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltnz8\" (UniqueName: \"kubernetes.io/projected/9f4c2084-9be1-422b-a365-fe3c08cd968c-kube-api-access-ltnz8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-zcx79\" (UID: \"9f4c2084-9be1-422b-a365-fe3c08cd968c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:15.757796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:15.757693 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:16.117920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:16.117852 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79"] Nov 03 13:14:16.120118 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:14:16.120086 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4c2084_9be1_422b_a365_fe3c08cd968c.slice/crio-53b25221818a4b55ec2f4354731719f9e94cf64484429ebbf554be3f0c37714b WatchSource:0}: Error finding container 53b25221818a4b55ec2f4354731719f9e94cf64484429ebbf554be3f0c37714b: Status 404 returned error can't find the container with id 53b25221818a4b55ec2f4354731719f9e94cf64484429ebbf554be3f0c37714b Nov 03 13:14:16.888930 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:14:16.888893 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:14:17.088507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:17.088414 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" event={"ID":"9f4c2084-9be1-422b-a365-fe3c08cd968c","Type":"ContainerStarted","Data":"6c929dcd38b416213032d94500da974a53675815d4f7088e63648ce2fd29b271"} Nov 03 13:14:17.088507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:17.088479 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" event={"ID":"9f4c2084-9be1-422b-a365-fe3c08cd968c","Type":"ContainerStarted","Data":"53b25221818a4b55ec2f4354731719f9e94cf64484429ebbf554be3f0c37714b"} Nov 03 13:14:17.199037 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:17.198986 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" podStartSLOduration=1.5819654760000001 podStartE2EDuration="2.198971238s" totalImagesPullingTime="617.005762ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:17.19788038 +0000 UTC m=+588.511852232" watchObservedRunningTime="2026-11-03 13:14:17.198971238 +0000 UTC m=+588.512943099" Nov 03 13:14:17.758624 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:17.758587 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:18.096992 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:18.096897 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:18.109438 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:18.109413 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:19.106899 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:19.106871 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8" Nov 03 13:14:20.080311 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.080278 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd"] Nov 03 13:14:20.082596 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.082573 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.185159 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.185117 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd"] Nov 03 13:14:20.185715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.185257 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 13:14:20.191877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.191743 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.191877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.191795 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pv5f\" (UniqueName: \"kubernetes.io/projected/9768d8a0-31ff-432d-bb2f-875f4077ea2a-kube-api-access-6pv5f\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.192058 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.191928 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.192058 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.191981 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.192058 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.192043 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.192214 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.192094 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9768d8a0-31ff-432d-bb2f-875f4077ea2a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.293287 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.293246 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.293287 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.293287 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pv5f\" (UniqueName: \"kubernetes.io/projected/9768d8a0-31ff-432d-bb2f-875f4077ea2a-kube-api-access-6pv5f\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.293561 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.293320 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.293561 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.293350 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.293561 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.293385 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.293561 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.293412 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9768d8a0-31ff-432d-bb2f-875f4077ea2a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.293762 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.293726 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.293903 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.293812 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.293903 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.293835 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.295540 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.295523 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9768d8a0-31ff-432d-bb2f-875f4077ea2a-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.295842 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.295825 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9768d8a0-31ff-432d-bb2f-875f4077ea2a-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.347516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.347449 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pv5f\" (UniqueName: \"kubernetes.io/projected/9768d8a0-31ff-432d-bb2f-875f4077ea2a-kube-api-access-6pv5f\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rthrd\" (UID: \"9768d8a0-31ff-432d-bb2f-875f4077ea2a\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.396656 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.396620 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:20.762670 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:20.762489 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd"] Nov 03 13:14:20.764660 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:14:20.764624 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9768d8a0_31ff_432d_bb2f_875f4077ea2a.slice/crio-96f33f8b26b0dd4d122f56e92edbe8ab741e999b6ff8e6de6f0e8993ff70146e WatchSource:0}: Error finding container 96f33f8b26b0dd4d122f56e92edbe8ab741e999b6ff8e6de6f0e8993ff70146e: Status 404 returned error can't find the container with id 96f33f8b26b0dd4d122f56e92edbe8ab741e999b6ff8e6de6f0e8993ff70146e Nov 03 13:14:21.112036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:21.111942 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" event={"ID":"9768d8a0-31ff-432d-bb2f-875f4077ea2a","Type":"ContainerStarted","Data":"96f33f8b26b0dd4d122f56e92edbe8ab741e999b6ff8e6de6f0e8993ff70146e"} Nov 03 13:14:22.117011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:22.116911 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" event={"ID":"9768d8a0-31ff-432d-bb2f-875f4077ea2a","Type":"ContainerStarted","Data":"c5264d874ff01b559113464cbbd8558035ea0509430015679bd8524b48aff433"} Nov 03 13:14:22.230530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:22.230476 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" podStartSLOduration=1.610942708 podStartE2EDuration="2.230461088s" totalImagesPullingTime="619.51838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:22.228803163 +0000 UTC m=+593.542775017" watchObservedRunningTime="2026-11-03 13:14:22.230461088 +0000 UTC m=+593.544432949" Nov 03 13:14:22.397066 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:22.396980 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:24.963753 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:24.963718 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv"] Nov 03 13:14:24.965825 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:24.965808 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.015277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.015241 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 13:14:25.035585 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.035553 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.035617 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987205e0-22af-461b-9c75-ec19537ebaa3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.035690 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.035873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.035753 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.035873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.035798 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.035873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.035818 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjl7j\" (UniqueName: \"kubernetes.io/projected/987205e0-22af-461b-9c75-ec19537ebaa3-kube-api-access-jjl7j\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.063587 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.063551 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv"] Nov 03 13:14:25.136970 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.136937 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.137161 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.136991 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987205e0-22af-461b-9c75-ec19537ebaa3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.137161 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.137012 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.137161 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.137054 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.137161 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.137080 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.137161 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.137107 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjl7j\" (UniqueName: \"kubernetes.io/projected/987205e0-22af-461b-9c75-ec19537ebaa3-kube-api-access-jjl7j\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.137428 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.137408 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.137528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.137488 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.137528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.137502 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.139261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.139240 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987205e0-22af-461b-9c75-ec19537ebaa3-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.139601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.139583 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987205e0-22af-461b-9c75-ec19537ebaa3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.191713 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.191672 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjl7j\" (UniqueName: \"kubernetes.io/projected/987205e0-22af-461b-9c75-ec19537ebaa3-kube-api-access-jjl7j\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tv7mv\" (UID: \"987205e0-22af-461b-9c75-ec19537ebaa3\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.277710 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.277623 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:25.642302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.642247 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv"] Nov 03 13:14:25.644830 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:14:25.644799 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987205e0_22af_461b_9c75_ec19537ebaa3.slice/crio-cde3639d9939657158e8477686c732bb83ace500c2e325becd7d0250d21549e6 WatchSource:0}: Error finding container cde3639d9939657158e8477686c732bb83ace500c2e325becd7d0250d21549e6: Status 404 returned error can't find the container with id cde3639d9939657158e8477686c732bb83ace500c2e325becd7d0250d21549e6 Nov 03 13:14:25.758923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.758885 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:25.771798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:25.771754 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:26.134740 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:26.134701 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" event={"ID":"987205e0-22af-461b-9c75-ec19537ebaa3","Type":"ContainerStarted","Data":"cde3639d9939657158e8477686c732bb83ace500c2e325becd7d0250d21549e6"} Nov 03 13:14:26.146556 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:26.146529 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-zcx79" Nov 03 13:14:27.141022 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:27.140986 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" event={"ID":"987205e0-22af-461b-9c75-ec19537ebaa3","Type":"ContainerStarted","Data":"30553e384686a3c98ea84779b140ffb88436f24356c10f66240364c9c4cc768a"} Nov 03 13:14:27.254602 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:27.254428 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" podStartSLOduration=2.669567093 podStartE2EDuration="3.254411745s" totalImagesPullingTime="584.844652ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:27.25375912 +0000 UTC m=+598.567731003" watchObservedRunningTime="2026-11-03 13:14:27.254411745 +0000 UTC m=+598.568383606" Nov 03 13:14:27.278749 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:27.278711 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:28.937570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:28.937541 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:14:28.938153 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:28.937826 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:14:30.397578 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:30.397547 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:30.410292 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:30.410261 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:31.167377 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:31.167349 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rthrd" Nov 03 13:14:34.081257 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.081221 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6xft2"] Nov 03 13:14:34.084359 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.084335 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.134594 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.134562 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 13:14:34.182852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.182812 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6xft2"] Nov 03 13:14:34.236055 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.236015 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.236241 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.236087 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.236241 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.236182 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0474dd31-37ce-4eb3-9c8c-336d9a19953b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.236241 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.236230 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-home\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.236371 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.236251 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bzf\" (UniqueName: \"kubernetes.io/projected/0474dd31-37ce-4eb3-9c8c-336d9a19953b-kube-api-access-n4bzf\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.236371 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.236320 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.337013 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.336932 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.337013 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.336977 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0474dd31-37ce-4eb3-9c8c-336d9a19953b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.337244 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.337110 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-home\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.337244 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.337157 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bzf\" (UniqueName: \"kubernetes.io/projected/0474dd31-37ce-4eb3-9c8c-336d9a19953b-kube-api-access-n4bzf\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.337244 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.337190 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.337244 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.337228 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.337607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.337580 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-home\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.337700 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.337636 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.337700 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.337650 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.339308 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.339279 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0474dd31-37ce-4eb3-9c8c-336d9a19953b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.339404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.339390 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0474dd31-37ce-4eb3-9c8c-336d9a19953b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.394962 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.394921 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bzf\" (UniqueName: \"kubernetes.io/projected/0474dd31-37ce-4eb3-9c8c-336d9a19953b-kube-api-access-n4bzf\") pod \"echo-simulator-kserve-64b6f8f96c-6xft2\" (UID: \"0474dd31-37ce-4eb3-9c8c-336d9a19953b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.395109 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.395095 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:34.798648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.798620 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-6xft2"] Nov 03 13:14:34.801295 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:14:34.801266 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0474dd31_37ce_4eb3_9c8c_336d9a19953b.slice/crio-deee8027043b5f75c561db69f8da4e6b6ea8014a5079d2c456abb63c8d814fc1 WatchSource:0}: Error finding container deee8027043b5f75c561db69f8da4e6b6ea8014a5079d2c456abb63c8d814fc1: Status 404 returned error can't find the container with id deee8027043b5f75c561db69f8da4e6b6ea8014a5079d2c456abb63c8d814fc1 Nov 03 13:14:34.803115 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:34.803090 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:14:35.174577 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:35.174541 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" event={"ID":"0474dd31-37ce-4eb3-9c8c-336d9a19953b","Type":"ContainerStarted","Data":"deee8027043b5f75c561db69f8da4e6b6ea8014a5079d2c456abb63c8d814fc1"} Nov 03 13:14:35.278494 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:35.278461 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:35.290635 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:35.290601 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:36.194960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:36.194710 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tv7mv" Nov 03 13:14:38.191330 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:38.191293 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" event={"ID":"0474dd31-37ce-4eb3-9c8c-336d9a19953b","Type":"ContainerStarted","Data":"57bbe37bf51065aa2ed41f582513ac85f5ab32784025db023c5ac8dec33c6ab8"} Nov 03 13:14:38.310470 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:38.310415 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" podStartSLOduration=1.842976677 podStartE2EDuration="4.310396431s" totalImagesPullingTime="2.467419754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:38.308400838 +0000 UTC m=+609.622372700" watchObservedRunningTime="2026-11-03 13:14:38.310396431 +0000 UTC m=+609.624368290" Nov 03 13:14:38.396224 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:38.396186 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:40.416086 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.416053 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5"] Nov 03 13:14:40.419578 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.419559 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.473061 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.473029 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 13:14:40.519439 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.519405 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5"] Nov 03 13:14:40.600576 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.600534 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.600758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.600587 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.600758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.600714 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.600758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.600750 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjxp5\" (UniqueName: \"kubernetes.io/projected/e961fb1a-50db-4f9b-9a17-b690246f59a7-kube-api-access-pjxp5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.600978 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.600877 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.600978 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.600931 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e961fb1a-50db-4f9b-9a17-b690246f59a7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.702302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.702218 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e961fb1a-50db-4f9b-9a17-b690246f59a7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.702302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.702282 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.702302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.702299 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.702562 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.702339 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.702562 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.702362 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjxp5\" (UniqueName: \"kubernetes.io/projected/e961fb1a-50db-4f9b-9a17-b690246f59a7-kube-api-access-pjxp5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.702562 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.702429 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.702845 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.702816 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.702845 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.702836 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.702976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.702858 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.704852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.704821 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e961fb1a-50db-4f9b-9a17-b690246f59a7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.704942 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.704865 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e961fb1a-50db-4f9b-9a17-b690246f59a7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:40.756992 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:40.756955 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjxp5\" (UniqueName: \"kubernetes.io/projected/e961fb1a-50db-4f9b-9a17-b690246f59a7-kube-api-access-pjxp5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5\" (UID: \"e961fb1a-50db-4f9b-9a17-b690246f59a7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:41.031442 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.031355 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:41.127032 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.126997 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt"] Nov 03 13:14:41.131364 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.131341 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.180986 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.180951 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 13:14:41.181174 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.181060 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wd2hr\"" Nov 03 13:14:41.229605 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.228017 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt"] Nov 03 13:14:41.308001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.307912 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7096b-66b8-4e64-aec0-fa83816adbab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.308001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.307957 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.308001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.307981 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.308300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.308097 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.308300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.308149 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.308300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.308192 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.308300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.308231 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.308300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.308269 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9pr\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-kube-api-access-ds9pr\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.308473 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.308388 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.394897 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.394869 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5"] Nov 03 13:14:41.396477 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:14:41.396449 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode961fb1a_50db_4f9b_9a17_b690246f59a7.slice/crio-2b57073e6fcf4509c6185802da80dfc95d9ea7be67937395aa01d0b52824faec WatchSource:0}: Error finding container 2b57073e6fcf4509c6185802da80dfc95d9ea7be67937395aa01d0b52824faec: Status 404 returned error can't find the container with id 2b57073e6fcf4509c6185802da80dfc95d9ea7be67937395aa01d0b52824faec Nov 03 13:14:41.409792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.409753 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.409916 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.409893 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.409966 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.409921 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.409966 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.409951 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.410056 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.409993 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.410056 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.410022 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.410175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.410048 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9pr\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-kube-api-access-ds9pr\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.410175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.410113 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.410175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.410162 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7096b-66b8-4e64-aec0-fa83816adbab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.410316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.410156 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.410495 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.410473 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.410583 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.410496 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.410651 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.410621 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.411121 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.411099 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7096b-66b8-4e64-aec0-fa83816adbab-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.412362 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.412339 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.412851 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.412827 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.464033 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.463995 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.464384 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.464208 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9pr\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-kube-api-access-ds9pr\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-bcrrt\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:41.744923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:41.744887 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:42.168416 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:42.168385 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt"] Nov 03 13:14:42.170144 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:14:42.170114 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c7096b_66b8_4e64_aec0_fa83816adbab.slice/crio-271b83388283f7d5b3c9cdf38351900ce50da5e35984fc0c5c78995f8c14e3e4 WatchSource:0}: Error finding container 271b83388283f7d5b3c9cdf38351900ce50da5e35984fc0c5c78995f8c14e3e4: Status 404 returned error can't find the container with id 271b83388283f7d5b3c9cdf38351900ce50da5e35984fc0c5c78995f8c14e3e4 Nov 03 13:14:42.214976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:42.214934 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" event={"ID":"e961fb1a-50db-4f9b-9a17-b690246f59a7","Type":"ContainerStarted","Data":"537c9b6179ac181fceb22d63f26e66ce211e7cfac9e63ea9a2c8b12499b0a087"} Nov 03 13:14:42.215159 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:42.214984 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" event={"ID":"e961fb1a-50db-4f9b-9a17-b690246f59a7","Type":"ContainerStarted","Data":"2b57073e6fcf4509c6185802da80dfc95d9ea7be67937395aa01d0b52824faec"} Nov 03 13:14:42.216280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:42.216239 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" event={"ID":"a5c7096b-66b8-4e64-aec0-fa83816adbab","Type":"ContainerStarted","Data":"271b83388283f7d5b3c9cdf38351900ce50da5e35984fc0c5c78995f8c14e3e4"} Nov 03 13:14:42.328762 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:42.328657 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" podStartSLOduration=1.7040417159999999 podStartE2EDuration="2.328642115s" totalImagesPullingTime="624.600399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:42.327170848 +0000 UTC m=+613.641142698" watchObservedRunningTime="2026-11-03 13:14:42.328642115 +0000 UTC m=+613.642613975" Nov 03 13:14:43.031626 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:43.031587 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:44.395841 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:44.395793 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:44.410338 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:44.410309 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:45.249422 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:45.249383 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-6xft2" Nov 03 13:14:45.986478 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:45.986444 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:14:45.986764 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:45.986512 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:14:45.986764 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:45.986542 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:14:46.237624 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:46.237527 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" event={"ID":"a5c7096b-66b8-4e64-aec0-fa83816adbab","Type":"ContainerStarted","Data":"797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc"} Nov 03 13:14:46.349589 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:46.349531 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podStartSLOduration=1.536448466 podStartE2EDuration="5.349515572s" totalImagesPullingTime="3.813067106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:46.347965392 +0000 UTC m=+617.661937274" watchObservedRunningTime="2026-11-03 13:14:46.349515572 +0000 UTC m=+617.663487431" Nov 03 13:14:46.752794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:46.748871 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:46.889786 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:14:46.889738 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:14:47.749796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:47.748696 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 13:14:47.749796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:47.748764 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 13:14:48.753795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:48.748195 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 13:14:48.753795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:48.748274 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 13:14:49.753796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:49.751183 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 13:14:49.753796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:49.751248 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 13:14:50.752807 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:50.752209 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 13:14:50.752807 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:50.752301 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 13:14:51.032247 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:51.032162 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:51.069748 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:51.069717 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:51.290856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:51.290757 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" Nov 03 13:14:51.755795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:51.755198 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 13:14:51.755795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:51.755266 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 13:14:52.745915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:52.745875 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 03 13:14:52.746284 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:52.745948 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 03 13:14:54.746662 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:54.746608 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:14:54.747216 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:54.746683 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:14:55.788952 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:55.788904 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:14:55.789474 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:55.788960 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:14:56.196866 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.196837 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:56.197233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.197204 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:56.197792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.197750 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:14:56.294634 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.294603 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc"] Nov 03 13:14:56.299296 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.299274 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.346683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.346650 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.346683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.346684 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.346894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.346760 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.346964 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.346944 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.347000 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.346985 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4srt\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-kube-api-access-t4srt\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.347052 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.347036 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.347123 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.347110 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.347211 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.347196 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.347257 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.347228 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.402685 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.402645 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc"] Nov 03 13:14:56.447791 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.447678 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.447791 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.447716 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.447791 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.447739 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.447859 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.447880 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.447897 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.447923 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.447945 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4srt\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-kube-api-access-t4srt\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.447976 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448386 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.448256 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448445 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.448406 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448550 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.448530 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.448754 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.448556 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.449319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.449294 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.455470 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.455448 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.455630 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.455612 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.502334 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.502296 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.502509 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.502428 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4srt\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-kube-api-access-t4srt\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-cwzrc\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:56.613077 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:56.613038 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:57.024422 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:57.024394 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc"] Nov 03 13:14:57.027167 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:14:57.027138 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91603f00_f1dd_4367_9d41_27c3dcb09ac7.slice/crio-e3154e43fa5cc1a66f1e5c0092a036f19f07684b51a6c70923e44ac18e7ace9e WatchSource:0}: Error finding container e3154e43fa5cc1a66f1e5c0092a036f19f07684b51a6c70923e44ac18e7ace9e: Status 404 returned error can't find the container with id e3154e43fa5cc1a66f1e5c0092a036f19f07684b51a6c70923e44ac18e7ace9e Nov 03 13:14:57.030142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:57.030110 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:14:57.030209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:57.030181 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:14:57.030249 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:57.030211 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:14:57.310324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:57.310223 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" event={"ID":"91603f00-f1dd-4367-9d41-27c3dcb09ac7","Type":"ContainerStarted","Data":"60458ff7d58b30c51803cc59ad0082bd4c3f53efc418348f14790362044cc20b"} Nov 03 13:14:57.310324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:57.310285 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" event={"ID":"91603f00-f1dd-4367-9d41-27c3dcb09ac7","Type":"ContainerStarted","Data":"e3154e43fa5cc1a66f1e5c0092a036f19f07684b51a6c70923e44ac18e7ace9e"} Nov 03 13:14:57.420663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:57.420607 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" podStartSLOduration=1.4205936129999999 podStartE2EDuration="1.420593613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:57.420240974 +0000 UTC m=+628.734212833" watchObservedRunningTime="2026-11-03 13:14:57.420593613 +0000 UTC m=+628.734565593" Nov 03 13:14:57.614166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:57.614078 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:14:58.614846 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:58.614802 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:14:58.615239 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:58.614867 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 13:14:59.614103 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:59.614062 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:14:59.614277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:14:59.614132 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 13:15:00.614316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:00.614274 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:15:00.614712 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:00.614337 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 13:15:01.614315 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:01.614278 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:15:01.614712 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:01.614331 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 13:15:02.613456 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:02.613420 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:15:02.613644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:02.613474 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 13:15:03.613642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:03.613606 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:15:03.614146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:03.613667 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 03 13:15:05.375594 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:05.375551 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:15:05.376098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:05.376073 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:15:05.376973 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:05.376955 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:15:07.693958 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:07.693927 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f5d89c4f6-jtgnc"] Nov 03 13:15:07.697565 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:07.697547 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" Nov 03 13:15:07.746839 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:07.746806 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trxdj\" (UniqueName: \"kubernetes.io/projected/cef8280b-0119-4e61-94c5-5ddea15d4f04-kube-api-access-trxdj\") pod \"authorino-5f5d89c4f6-jtgnc\" (UID: \"cef8280b-0119-4e61-94c5-5ddea15d4f04\") " pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" Nov 03 13:15:07.746984 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:07.746873 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cef8280b-0119-4e61-94c5-5ddea15d4f04-tls-cert\") pod \"authorino-5f5d89c4f6-jtgnc\" (UID: \"cef8280b-0119-4e61-94c5-5ddea15d4f04\") " pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" Nov 03 13:15:07.749573 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:07.749543 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f5d89c4f6-jtgnc"] Nov 03 13:15:07.847297 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:07.847256 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trxdj\" (UniqueName: \"kubernetes.io/projected/cef8280b-0119-4e61-94c5-5ddea15d4f04-kube-api-access-trxdj\") pod \"authorino-5f5d89c4f6-jtgnc\" (UID: \"cef8280b-0119-4e61-94c5-5ddea15d4f04\") " pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" Nov 03 13:15:07.847481 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:07.847329 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cef8280b-0119-4e61-94c5-5ddea15d4f04-tls-cert\") pod \"authorino-5f5d89c4f6-jtgnc\" (UID: \"cef8280b-0119-4e61-94c5-5ddea15d4f04\") " pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" Nov 03 13:15:07.849987 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:07.849955 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cef8280b-0119-4e61-94c5-5ddea15d4f04-tls-cert\") pod \"authorino-5f5d89c4f6-jtgnc\" (UID: \"cef8280b-0119-4e61-94c5-5ddea15d4f04\") " pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" Nov 03 13:15:07.902571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:07.902541 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trxdj\" (UniqueName: \"kubernetes.io/projected/cef8280b-0119-4e61-94c5-5ddea15d4f04-kube-api-access-trxdj\") pod \"authorino-5f5d89c4f6-jtgnc\" (UID: \"cef8280b-0119-4e61-94c5-5ddea15d4f04\") " pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" Nov 03 13:15:08.008973 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:08.008877 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" Nov 03 13:15:08.384140 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:08.384105 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f5d89c4f6-jtgnc"] Nov 03 13:15:08.385796 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:15:08.385753 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcef8280b_0119_4e61_94c5_5ddea15d4f04.slice/crio-e7e108f2c289fc568c4506d30df62fd7cc7cc0e1477c252fb895affbfb0c6447 WatchSource:0}: Error finding container e7e108f2c289fc568c4506d30df62fd7cc7cc0e1477c252fb895affbfb0c6447: Status 404 returned error can't find the container with id e7e108f2c289fc568c4506d30df62fd7cc7cc0e1477c252fb895affbfb0c6447 Nov 03 13:15:09.364293 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:09.364247 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" event={"ID":"cef8280b-0119-4e61-94c5-5ddea15d4f04","Type":"ContainerStarted","Data":"e7e108f2c289fc568c4506d30df62fd7cc7cc0e1477c252fb895affbfb0c6447"} Nov 03 13:15:10.370384 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.370284 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" event={"ID":"cef8280b-0119-4e61-94c5-5ddea15d4f04","Type":"ContainerStarted","Data":"ee4252d9f15a429e560b21de46bb0a4477bbeab9ea74b6a54f0951616c13e79d"} Nov 03 13:15:10.481287 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.481234 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f5d89c4f6-jtgnc" podStartSLOduration=1.8416375249999999 podStartE2EDuration="3.48121646s" totalImagesPullingTime="1.639578935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:10.48081108 +0000 UTC m=+641.794782939" watchObservedRunningTime="2026-11-03 13:15:10.48121646 +0000 UTC m=+641.795188320" Nov 03 13:15:10.530273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.530232 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65bf5b6f59-2k52j"] Nov 03 13:15:10.530476 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.530449 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" podUID="f97b32d7-7d37-4f72-a40e-5aea48789ff5" containerName="authorino" containerID="cri-o://6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec" gracePeriod=30 Nov 03 13:15:10.793693 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.793667 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:15:10.875406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.875374 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97b32d7-7d37-4f72-a40e-5aea48789ff5-kube-api-access-z9sm5\" (UniqueName: \"kubernetes.io/projected/f97b32d7-7d37-4f72-a40e-5aea48789ff5-kube-api-access-z9sm5\") pod \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\" (UID: \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\") " Nov 03 13:15:10.875566 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.875442 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f97b32d7-7d37-4f72-a40e-5aea48789ff5-tls-cert\" (UniqueName: \"kubernetes.io/secret/f97b32d7-7d37-4f72-a40e-5aea48789ff5-tls-cert\") pod \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\" (UID: \"f97b32d7-7d37-4f72-a40e-5aea48789ff5\") " Nov 03 13:15:10.877515 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.877482 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97b32d7-7d37-4f72-a40e-5aea48789ff5-kube-api-access-z9sm5" pod "f97b32d7-7d37-4f72-a40e-5aea48789ff5" (UID: "f97b32d7-7d37-4f72-a40e-5aea48789ff5"). InnerVolumeSpecName "kube-api-access-z9sm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:15:10.886751 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.886715 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97b32d7-7d37-4f72-a40e-5aea48789ff5-tls-cert" pod "f97b32d7-7d37-4f72-a40e-5aea48789ff5" (UID: "f97b32d7-7d37-4f72-a40e-5aea48789ff5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:15:10.976417 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.976326 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9sm5\" (UniqueName: \"kubernetes.io/projected/f97b32d7-7d37-4f72-a40e-5aea48789ff5-kube-api-access-z9sm5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:10.976417 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:10.976358 4541 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f97b32d7-7d37-4f72-a40e-5aea48789ff5-tls-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:11.375938 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:11.375856 4541 generic.go:356] "Generic (PLEG): container finished" podID="f97b32d7-7d37-4f72-a40e-5aea48789ff5" containerID="6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec" exitCode=0 Nov 03 13:15:11.375938 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:11.375889 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" event={"ID":"f97b32d7-7d37-4f72-a40e-5aea48789ff5","Type":"ContainerDied","Data":"6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec"} Nov 03 13:15:11.375938 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:11.375930 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" event={"ID":"f97b32d7-7d37-4f72-a40e-5aea48789ff5","Type":"ContainerDied","Data":"0e06e970bff2b18a46d441cd55364a6337bf97d534e465e187ff67a2b67d7561"} Nov 03 13:15:11.376499 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:11.375947 4541 scope.go:122] "RemoveContainer" containerID="6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec" Nov 03 13:15:11.376499 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:11.375870 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65bf5b6f59-2k52j" Nov 03 13:15:11.386025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:11.386004 4541 scope.go:122] "RemoveContainer" containerID="6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec" Nov 03 13:15:11.386312 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:15:11.386291 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec\": container with ID starting with 6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec not found: ID does not exist" containerID="6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec" Nov 03 13:15:11.386407 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:11.386320 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec"} err="failed to get container status \"6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec\": rpc error: code = NotFound desc = could not find container \"6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec\": container with ID starting with 6b12fdc1583a80970760f0f19212918f5143ba63cad4e42265cf8ea0d89701ec not found: ID does not exist" Nov 03 13:15:11.538472 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:11.538435 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65bf5b6f59-2k52j"] Nov 03 13:15:11.585442 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:11.585399 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65bf5b6f59-2k52j"] Nov 03 13:15:12.866716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:12.866683 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97b32d7-7d37-4f72-a40e-5aea48789ff5" path="/var/lib/kubelet/pods/f97b32d7-7d37-4f72-a40e-5aea48789ff5/volumes" Nov 03 13:15:16.889734 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:15:16.889698 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:15:40.323131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.323037 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4"] Nov 03 13:15:40.323583 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.323570 4541 cpu_manager.go:414] "containerMap: removing container" podUID="f97b32d7-7d37-4f72-a40e-5aea48789ff5" containerName="authorino" Nov 03 13:15:40.323650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.323590 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97b32d7-7d37-4f72-a40e-5aea48789ff5" containerName="authorino" Nov 03 13:15:40.323710 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.323701 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97b32d7-7d37-4f72-a40e-5aea48789ff5" containerName="authorino" Nov 03 13:15:40.330839 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.330816 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.428977 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.428945 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4"] Nov 03 13:15:40.432665 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.432648 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-67349d1c-openshift-default-dockercfg-2sgd5\"" Nov 03 13:15:40.455157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.455112 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-podinfo\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.455157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.455159 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-certs\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.455350 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.455228 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-envoy\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.455350 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.455336 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hzh\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-kube-api-access-p7hzh\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.455425 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.455376 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-credential-socket\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.455425 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.455418 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-socket\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.455497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.455485 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c077100-2f32-4ff0-9e89-99b4523c1278-istiod-ca-cert\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.455547 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.455534 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-data\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.455583 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.455553 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-token\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.556551 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.556503 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-socket\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.556551 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.556555 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c077100-2f32-4ff0-9e89-99b4523c1278-istiod-ca-cert\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.556915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.556600 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-data\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.556915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.556617 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-token\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.556915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.556642 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-podinfo\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.556915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.556660 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-certs\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.556915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.556677 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-envoy\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.556915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.556721 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hzh\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-kube-api-access-p7hzh\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.556915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.556762 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-credential-socket\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.557276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.557002 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-socket\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.557276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.557138 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-data\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.557276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.557186 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-credential-socket\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.557276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.557253 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-certs\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.557488 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.557451 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c077100-2f32-4ff0-9e89-99b4523c1278-istiod-ca-cert\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.558955 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.558934 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-envoy\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.559151 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.559132 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-podinfo\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.615727 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.615652 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-token\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.623239 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.623208 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hzh\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-kube-api-access-p7hzh\") pod \"e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:40.643766 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:40.643734 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:41.072661 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.072632 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4"] Nov 03 13:15:41.074969 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:15:41.074942 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c077100_2f32_4ff0_9e89_99b4523c1278.slice/crio-402efebc258f0b85623cf883fe53567b2cc7d0d16370465abb30da8a645661c6 WatchSource:0}: Error finding container 402efebc258f0b85623cf883fe53567b2cc7d0d16370465abb30da8a645661c6: Status 404 returned error can't find the container with id 402efebc258f0b85623cf883fe53567b2cc7d0d16370465abb30da8a645661c6 Nov 03 13:15:41.077095 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.077063 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:15:41.077172 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.077131 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:15:41.077172 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.077161 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:15:41.321595 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.321559 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh"] Nov 03 13:15:41.325250 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.325192 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.377025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.376985 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-cb1bd0b2-gw-openshift-default-dockercfg-qwc97\"" Nov 03 13:15:41.386646 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.386615 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh"] Nov 03 13:15:41.464742 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.464709 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-credential-socket\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.464953 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.464754 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-podinfo\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.464953 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.464884 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hgw\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-kube-api-access-26hgw\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.464953 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.464938 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-token\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.465083 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.464968 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-data\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.465083 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.465066 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-envoy\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.465167 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.465091 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istiod-ca-cert\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.465167 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.465136 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-socket\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.465237 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.465194 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-certs\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.510352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.510315 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" event={"ID":"9c077100-2f32-4ff0-9e89-99b4523c1278","Type":"ContainerStarted","Data":"06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b"} Nov 03 13:15:41.510541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.510366 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" event={"ID":"9c077100-2f32-4ff0-9e89-99b4523c1278","Type":"ContainerStarted","Data":"402efebc258f0b85623cf883fe53567b2cc7d0d16370465abb30da8a645661c6"} Nov 03 13:15:41.566062 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566021 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-envoy\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566073 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istiod-ca-cert\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566126 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-socket\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566153 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-certs\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566211 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-credential-socket\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566454 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566242 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-podinfo\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566454 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566277 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hgw\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-kube-api-access-26hgw\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566454 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566307 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-token\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566454 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566345 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-data\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566684 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566643 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-socket\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.566991 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.566968 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-data\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.567260 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.567235 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-credential-socket\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.567387 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.567364 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-certs\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.567575 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.567556 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istiod-ca-cert\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.568422 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.568400 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-envoy\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.569225 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.569205 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-podinfo\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.620958 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.620883 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-token\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.621094 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.621015 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hgw\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-kube-api-access-26hgw\") pod \"e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.623623 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.623581 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" podStartSLOduration=1.62356738 podStartE2EDuration="1.62356738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:41.62267916 +0000 UTC m=+672.936651020" watchObservedRunningTime="2026-11-03 13:15:41.62356738 +0000 UTC m=+672.937539240" Nov 03 13:15:41.638134 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.638109 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:41.644270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:41.644238 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:42.052494 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:42.052466 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh"] Nov 03 13:15:42.053996 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:15:42.053964 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdff3960_b86d_4c10_b7d7_137f9fa9f3db.slice/crio-683396a4a0119bc594674e502fc7d12da2ea939c20b8832cc18d4083b234c56b WatchSource:0}: Error finding container 683396a4a0119bc594674e502fc7d12da2ea939c20b8832cc18d4083b234c56b: Status 404 returned error can't find the container with id 683396a4a0119bc594674e502fc7d12da2ea939c20b8832cc18d4083b234c56b Nov 03 13:15:42.056592 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:42.056561 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:15:42.056650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:42.056624 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:15:42.056699 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:42.056653 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:15:42.516493 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:42.516458 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" event={"ID":"fdff3960-b86d-4c10-b7d7-137f9fa9f3db","Type":"ContainerStarted","Data":"eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8"} Nov 03 13:15:42.516905 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:42.516520 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" event={"ID":"fdff3960-b86d-4c10-b7d7-137f9fa9f3db","Type":"ContainerStarted","Data":"683396a4a0119bc594674e502fc7d12da2ea939c20b8832cc18d4083b234c56b"} Nov 03 13:15:42.629011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:42.628954 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" podStartSLOduration=1.628938373 podStartE2EDuration="1.628938373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:42.627681327 +0000 UTC m=+673.941653186" watchObservedRunningTime="2026-11-03 13:15:42.628938373 +0000 UTC m=+673.942910232" Nov 03 13:15:42.638288 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:42.638241 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:42.649766 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:42.649740 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:43.521622 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:43.521589 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:43.522642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:43.522620 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:15:43.647321 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:43.647270 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:44.526248 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:44.526208 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:44.527062 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:44.527044 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:15:46.891078 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:15:46.891042 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:15:47.139761 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.139729 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7"] Nov 03 13:15:47.143316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.143253 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.196187 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.196131 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a7a9081d-openshift-default-dockercfg-9nrpm\"" Nov 03 13:15:47.218307 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.218267 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-certs\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.218307 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.218306 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ww2\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-kube-api-access-44ww2\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.218516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.218338 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-token\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.218516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.218413 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-credential-socket\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.218516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.218478 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-data\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.218516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.218514 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istiod-ca-cert\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.218700 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.218532 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-envoy\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.218700 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.218595 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-podinfo\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.218821 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.218730 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-socket\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.242877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.242812 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7"] Nov 03 13:15:47.319983 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.319950 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-certs\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.319983 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.319983 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ww2\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-kube-api-access-44ww2\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320005 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-token\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320043 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-credential-socket\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320085 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-data\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320138 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istiod-ca-cert\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320164 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-envoy\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320193 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-podinfo\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320222 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-socket\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320614 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320397 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-certs\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320614 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320460 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-credential-socket\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320614 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320500 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-data\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320815 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320675 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-socket\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.320960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.320938 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istiod-ca-cert\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.322444 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.322424 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-envoy\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.322746 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.322728 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-podinfo\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.366029 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.365997 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc"] Nov 03 13:15:47.369601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.369580 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:15:47.375751 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.375727 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-token\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.378413 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.378382 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ww2\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-kube-api-access-44ww2\") pod \"e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.420990 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.420959 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gd5d\" (UniqueName: \"kubernetes.io/projected/41040ec9-636c-47f7-a965-b9f7229deb2c-kube-api-access-9gd5d\") pod \"payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc\" (UID: \"41040ec9-636c-47f7-a965-b9f7229deb2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:15:47.421158 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.421005 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41040ec9-636c-47f7-a965-b9f7229deb2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc\" (UID: \"41040ec9-636c-47f7-a965-b9f7229deb2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:15:47.421338 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.421322 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67349d1c\"" Nov 03 13:15:47.428280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.428250 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-67349d1c-dockercfg-sh2cf\"" Nov 03 13:15:47.429021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.428998 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc"] Nov 03 13:15:47.456510 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.456458 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:47.522856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.521817 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gd5d\" (UniqueName: \"kubernetes.io/projected/41040ec9-636c-47f7-a965-b9f7229deb2c-kube-api-access-9gd5d\") pod \"payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc\" (UID: \"41040ec9-636c-47f7-a965-b9f7229deb2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:15:47.522856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.521866 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41040ec9-636c-47f7-a965-b9f7229deb2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc\" (UID: \"41040ec9-636c-47f7-a965-b9f7229deb2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:15:47.522856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.522786 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41040ec9-636c-47f7-a965-b9f7229deb2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc\" (UID: \"41040ec9-636c-47f7-a965-b9f7229deb2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:15:47.576273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.576231 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gd5d\" (UniqueName: \"kubernetes.io/projected/41040ec9-636c-47f7-a965-b9f7229deb2c-kube-api-access-9gd5d\") pod \"payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc\" (UID: \"41040ec9-636c-47f7-a965-b9f7229deb2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:15:47.697274 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.697187 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:15:47.884641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.884617 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7"] Nov 03 13:15:47.886648 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:15:47.886609 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd25c3f_f82a_4d42_aadb_29f6398f44f1.slice/crio-4add471b07cc9a02dfe1a09b5cf57f31cfab688dccc4d064334d10fe2ba4bef9 WatchSource:0}: Error finding container 4add471b07cc9a02dfe1a09b5cf57f31cfab688dccc4d064334d10fe2ba4bef9: Status 404 returned error can't find the container with id 4add471b07cc9a02dfe1a09b5cf57f31cfab688dccc4d064334d10fe2ba4bef9 Nov 03 13:15:47.888946 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.888914 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:15:47.889040 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.888974 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:15:47.889040 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:47.889004 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:15:48.062051 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:48.062026 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc"] Nov 03 13:15:48.064046 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:15:48.064013 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41040ec9_636c_47f7_a965_b9f7229deb2c.slice/crio-0c9ae3ea3ecfdc1d9e5bc7c0f7a20bb578bd8f0ef60f17504b312889112b1847 WatchSource:0}: Error finding container 0c9ae3ea3ecfdc1d9e5bc7c0f7a20bb578bd8f0ef60f17504b312889112b1847: Status 404 returned error can't find the container with id 0c9ae3ea3ecfdc1d9e5bc7c0f7a20bb578bd8f0ef60f17504b312889112b1847 Nov 03 13:15:48.551324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:48.551280 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" event={"ID":"7bd25c3f-f82a-4d42-aadb-29f6398f44f1","Type":"ContainerStarted","Data":"8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa"} Nov 03 13:15:48.551519 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:48.551386 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" event={"ID":"7bd25c3f-f82a-4d42-aadb-29f6398f44f1","Type":"ContainerStarted","Data":"4add471b07cc9a02dfe1a09b5cf57f31cfab688dccc4d064334d10fe2ba4bef9"} Nov 03 13:15:48.552897 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:48.552870 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" event={"ID":"41040ec9-636c-47f7-a965-b9f7229deb2c","Type":"ContainerStarted","Data":"d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed"} Nov 03 13:15:48.553018 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:48.552916 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" event={"ID":"41040ec9-636c-47f7-a965-b9f7229deb2c","Type":"ContainerStarted","Data":"0c9ae3ea3ecfdc1d9e5bc7c0f7a20bb578bd8f0ef60f17504b312889112b1847"} Nov 03 13:15:48.553085 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:48.553020 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:15:48.669827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:48.669762 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" podStartSLOduration=1.669748748 podStartE2EDuration="1.669748748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:48.667638204 +0000 UTC m=+679.981610065" watchObservedRunningTime="2026-11-03 13:15:48.669748748 +0000 UTC m=+679.983720608" Nov 03 13:15:48.777372 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:48.777318 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" podStartSLOduration=1.777303206 podStartE2EDuration="1.777303206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:48.776191598 +0000 UTC m=+680.090163458" watchObservedRunningTime="2026-11-03 13:15:48.777303206 +0000 UTC m=+680.091275062" Nov 03 13:15:49.457466 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:49.457426 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:49.462606 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:49.462582 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:49.557836 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:49.557799 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:49.558662 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:49.558643 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:15:50.338670 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.338635 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl"] Nov 03 13:15:50.342340 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.342320 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.345283 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.345261 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfx7f\" (UniqueName: \"kubernetes.io/projected/83fc22fa-3df0-469b-a570-72bba1b7ed38-kube-api-access-lfx7f\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.345342 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.345317 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-metrics-certs\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.345408 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.345392 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fc22fa-3df0-469b-a570-72bba1b7ed38-openshift-service-ca\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.345443 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.345436 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.394660 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.394623 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-821a13f6-serving-cert\"" Nov 03 13:15:50.445165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.445123 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl"] Nov 03 13:15:50.446130 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.446107 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fc22fa-3df0-469b-a570-72bba1b7ed38-openshift-service-ca\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.446248 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.446159 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.446248 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.446232 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfx7f\" (UniqueName: \"kubernetes.io/projected/83fc22fa-3df0-469b-a570-72bba1b7ed38-kube-api-access-lfx7f\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.446331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.446256 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-metrics-certs\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.446331 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:15:50.446306 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-serving-cert: secret "maas-api-e2e-ait-821a13f6-serving-cert" not found Nov 03 13:15:50.446894 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:15:50.446862 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls podName:83fc22fa-3df0-469b-a570-72bba1b7ed38 nodeName:}" failed. No retries permitted until 2026-11-03 13:15:50.946832982 +0000 UTC m=+682.260804824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls") pod "maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" (UID: "83fc22fa-3df0-469b-a570-72bba1b7ed38") : secret "maas-api-e2e-ait-821a13f6-serving-cert" not found Nov 03 13:15:50.447543 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.447522 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fc22fa-3df0-469b-a570-72bba1b7ed38-openshift-service-ca\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.454426 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.454406 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-metrics-certs\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.501590 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.501559 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfx7f\" (UniqueName: \"kubernetes.io/projected/83fc22fa-3df0-469b-a570-72bba1b7ed38-kube-api-access-lfx7f\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.949421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.949387 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.951998 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.951942 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls\") pod \"maas-api-e2e-ait-821a13f6-568d745c84-qz8sl\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:50.955373 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:50.955350 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:51.319165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:51.319139 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl"] Nov 03 13:15:51.321142 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:15:51.321111 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fc22fa_3df0_469b_a570_72bba1b7ed38.slice/crio-af3925df52d5dde4dd30aaec2551fe3ad56753ead6d727e2c0ea2c27baf906c1 WatchSource:0}: Error finding container af3925df52d5dde4dd30aaec2551fe3ad56753ead6d727e2c0ea2c27baf906c1: Status 404 returned error can't find the container with id af3925df52d5dde4dd30aaec2551fe3ad56753ead6d727e2c0ea2c27baf906c1 Nov 03 13:15:51.566733 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:51.566700 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" event={"ID":"83fc22fa-3df0-469b-a570-72bba1b7ed38","Type":"ContainerStarted","Data":"af3925df52d5dde4dd30aaec2551fe3ad56753ead6d727e2c0ea2c27baf906c1"} Nov 03 13:15:52.573131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:52.573036 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" event={"ID":"83fc22fa-3df0-469b-a570-72bba1b7ed38","Type":"ContainerStarted","Data":"b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822"} Nov 03 13:15:52.573499 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:52.573150 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:52.682937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:52.682884 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" podStartSLOduration=1.682475573 podStartE2EDuration="2.682867911s" totalImagesPullingTime="1.000392338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:52.681590024 +0000 UTC m=+683.995561885" watchObservedRunningTime="2026-11-03 13:15:52.682867911 +0000 UTC m=+683.996839770" Nov 03 13:15:54.908616 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:54.908566 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck"] Nov 03 13:15:54.912273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:54.912249 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:15:54.968609 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:54.968578 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck"] Nov 03 13:15:54.988322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:54.988292 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p4p6\" (UniqueName: \"kubernetes.io/projected/d33a161b-a360-465a-94ab-9e9d74a30318-kube-api-access-9p4p6\") pod \"payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck\" (UID: \"d33a161b-a360-465a-94ab-9e9d74a30318\") " pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:15:54.988480 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:54.988445 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33a161b-a360-465a-94ab-9e9d74a30318-plugins-config-volume\") pod \"payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck\" (UID: \"d33a161b-a360-465a-94ab-9e9d74a30318\") " pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:15:55.010096 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.010068 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-11ca03ad\"" Nov 03 13:15:55.010096 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.010081 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-11ca03ad-dockercfg-c9c6x\"" Nov 03 13:15:55.089236 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.089201 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p4p6\" (UniqueName: \"kubernetes.io/projected/d33a161b-a360-465a-94ab-9e9d74a30318-kube-api-access-9p4p6\") pod \"payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck\" (UID: \"d33a161b-a360-465a-94ab-9e9d74a30318\") " pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:15:55.089411 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.089307 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33a161b-a360-465a-94ab-9e9d74a30318-plugins-config-volume\") pod \"payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck\" (UID: \"d33a161b-a360-465a-94ab-9e9d74a30318\") " pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:15:55.090050 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.090025 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33a161b-a360-465a-94ab-9e9d74a30318-plugins-config-volume\") pod \"payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck\" (UID: \"d33a161b-a360-465a-94ab-9e9d74a30318\") " pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:15:55.142930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.142891 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p4p6\" (UniqueName: \"kubernetes.io/projected/d33a161b-a360-465a-94ab-9e9d74a30318-kube-api-access-9p4p6\") pod \"payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck\" (UID: \"d33a161b-a360-465a-94ab-9e9d74a30318\") " pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:15:55.224309 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.224217 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:15:55.595906 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.595867 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck"] Nov 03 13:15:55.597183 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:15:55.597135 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33a161b_a360_465a_94ab_9e9d74a30318.slice/crio-450469d6f6cc8ed88068ec08b44a0de917545d77ce2017bcb912adf15e6ff7b8 WatchSource:0}: Error finding container 450469d6f6cc8ed88068ec08b44a0de917545d77ce2017bcb912adf15e6ff7b8: Status 404 returned error can't find the container with id 450469d6f6cc8ed88068ec08b44a0de917545d77ce2017bcb912adf15e6ff7b8 Nov 03 13:15:55.914880 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.914852 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv"] Nov 03 13:15:55.918309 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.918290 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:55.972842 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.972812 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv"] Nov 03 13:15:55.977822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:55.977797 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-67349d1c-serving-cert\"" Nov 03 13:15:56.008888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.008849 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70059faa-8450-4c6c-ba88-b9f452eed7ce-openshift-service-ca\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.009046 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.008892 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-metrics-certs\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.009046 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.008997 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.009122 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.009089 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlxcn\" (UniqueName: \"kubernetes.io/projected/70059faa-8450-4c6c-ba88-b9f452eed7ce-kube-api-access-wlxcn\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.109625 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.109594 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlxcn\" (UniqueName: \"kubernetes.io/projected/70059faa-8450-4c6c-ba88-b9f452eed7ce-kube-api-access-wlxcn\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.109856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.109688 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70059faa-8450-4c6c-ba88-b9f452eed7ce-openshift-service-ca\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.109856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.109718 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-metrics-certs\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.109856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.109758 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.110035 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:15:56.109919 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-serving-cert: secret "maas-api-e2e-ait-67349d1c-serving-cert" not found Nov 03 13:15:56.110035 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:15:56.110001 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls podName:70059faa-8450-4c6c-ba88-b9f452eed7ce nodeName:}" failed. No retries permitted until 2026-11-03 13:15:56.609979567 +0000 UTC m=+687.923951423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls") pod "maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" (UID: "70059faa-8450-4c6c-ba88-b9f452eed7ce") : secret "maas-api-e2e-ait-67349d1c-serving-cert" not found Nov 03 13:15:56.110418 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.110392 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70059faa-8450-4c6c-ba88-b9f452eed7ce-openshift-service-ca\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.112374 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.112349 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-metrics-certs\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.169553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.169456 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlxcn\" (UniqueName: \"kubernetes.io/projected/70059faa-8450-4c6c-ba88-b9f452eed7ce-kube-api-access-wlxcn\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.591444 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.591345 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" event={"ID":"d33a161b-a360-465a-94ab-9e9d74a30318","Type":"ContainerStarted","Data":"cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603"} Nov 03 13:15:56.591444 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.591400 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" event={"ID":"d33a161b-a360-465a-94ab-9e9d74a30318","Type":"ContainerStarted","Data":"450469d6f6cc8ed88068ec08b44a0de917545d77ce2017bcb912adf15e6ff7b8"} Nov 03 13:15:56.591673 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.591490 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:15:56.615115 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.615082 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.617478 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.617455 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls\") pod \"maas-api-e2e-ait-67349d1c-74746c9c-c8pxv\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:56.700056 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.699998 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" podStartSLOduration=2.699985078 podStartE2EDuration="2.699985078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:56.698003388 +0000 UTC m=+688.011975250" watchObservedRunningTime="2026-11-03 13:15:56.699985078 +0000 UTC m=+688.013956937" Nov 03 13:15:56.831122 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:56.831088 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:57.206120 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:57.205353 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv"] Nov 03 13:15:57.209921 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:15:57.209888 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70059faa_8450_4c6c_ba88_b9f452eed7ce.slice/crio-71950ecff91dc6de021c62b4a93a6bbe8235716a7fa6e378b87df90957fe70aa WatchSource:0}: Error finding container 71950ecff91dc6de021c62b4a93a6bbe8235716a7fa6e378b87df90957fe70aa: Status 404 returned error can't find the container with id 71950ecff91dc6de021c62b4a93a6bbe8235716a7fa6e378b87df90957fe70aa Nov 03 13:15:57.597177 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:57.597081 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" event={"ID":"70059faa-8450-4c6c-ba88-b9f452eed7ce","Type":"ContainerStarted","Data":"71950ecff91dc6de021c62b4a93a6bbe8235716a7fa6e378b87df90957fe70aa"} Nov 03 13:15:58.581947 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:58.581916 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:15:58.602895 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:58.602859 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" event={"ID":"70059faa-8450-4c6c-ba88-b9f452eed7ce","Type":"ContainerStarted","Data":"e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228"} Nov 03 13:15:58.603023 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:58.602963 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:15:58.800894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:58.800831 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" podStartSLOduration=2.929916584 podStartE2EDuration="3.800810166s" totalImagesPullingTime="870.893582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:58.800150354 +0000 UTC m=+690.114122217" watchObservedRunningTime="2026-11-03 13:15:58.800810166 +0000 UTC m=+690.114782029" Nov 03 13:15:59.560445 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:15:59.560410 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:16:01.245276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.245245 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf"] Nov 03 13:16:01.249166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.249145 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.298354 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.298322 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cb1bd0b2-serving-cert\"" Nov 03 13:16:01.347322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.347281 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf"] Nov 03 13:16:01.364103 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.364065 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c5f500-e16e-4b94-8627-0fc76293c20a-openshift-service-ca\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.364277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.364124 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-metrics-certs\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.364277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.364217 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.364378 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.364359 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh6d7\" (UniqueName: \"kubernetes.io/projected/25c5f500-e16e-4b94-8627-0fc76293c20a-kube-api-access-jh6d7\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.465482 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.465440 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh6d7\" (UniqueName: \"kubernetes.io/projected/25c5f500-e16e-4b94-8627-0fc76293c20a-kube-api-access-jh6d7\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.465650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.465492 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c5f500-e16e-4b94-8627-0fc76293c20a-openshift-service-ca\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.465650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.465521 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-metrics-certs\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.465650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.465543 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.465650 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:16:01.465639 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-serving-cert: secret "maas-api-e2e-migrate-cb1bd0b2-serving-cert" not found Nov 03 13:16:01.465929 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:16:01.465698 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls podName:25c5f500-e16e-4b94-8627-0fc76293c20a nodeName:}" failed. No retries permitted until 2026-11-03 13:16:01.965681479 +0000 UTC m=+693.279653333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls") pod "maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" (UID: "25c5f500-e16e-4b94-8627-0fc76293c20a") : secret "maas-api-e2e-migrate-cb1bd0b2-serving-cert" not found Nov 03 13:16:01.466284 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.466255 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c5f500-e16e-4b94-8627-0fc76293c20a-openshift-service-ca\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.468116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.468092 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-metrics-certs\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.520222 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.520136 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh6d7\" (UniqueName: \"kubernetes.io/projected/25c5f500-e16e-4b94-8627-0fc76293c20a-kube-api-access-jh6d7\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.970827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.970791 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:01.973230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:01.973196 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls\") pod \"maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:02.161400 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:02.161368 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:02.525554 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:02.525521 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf"] Nov 03 13:16:02.527715 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:16:02.527680 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c5f500_e16e_4b94_8627_0fc76293c20a.slice/crio-f1f2a891d6c2bca537f3439feab1ad5f8804467667b5aa66d77b0309a8dd43e0 WatchSource:0}: Error finding container f1f2a891d6c2bca537f3439feab1ad5f8804467667b5aa66d77b0309a8dd43e0: Status 404 returned error can't find the container with id f1f2a891d6c2bca537f3439feab1ad5f8804467667b5aa66d77b0309a8dd43e0 Nov 03 13:16:02.626212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:02.626175 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" event={"ID":"25c5f500-e16e-4b94-8627-0fc76293c20a","Type":"ContainerStarted","Data":"f1f2a891d6c2bca537f3439feab1ad5f8804467667b5aa66d77b0309a8dd43e0"} Nov 03 13:16:03.632564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:03.632528 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" event={"ID":"25c5f500-e16e-4b94-8627-0fc76293c20a","Type":"ContainerStarted","Data":"83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80"} Nov 03 13:16:03.632996 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:03.632593 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:03.742608 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:03.742512 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" podStartSLOduration=1.855494107 podStartE2EDuration="2.742496619s" totalImagesPullingTime="887.002512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:03.740403346 +0000 UTC m=+695.054375205" watchObservedRunningTime="2026-11-03 13:16:03.742496619 +0000 UTC m=+695.056468480" Nov 03 13:16:04.613984 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:04.613956 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:16:07.611985 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:07.611950 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:16:09.670797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:09.669943 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:11.434065 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.434026 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh"] Nov 03 13:16:11.451516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.451482 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.491913 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.491885 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh"] Nov 03 13:16:11.582939 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.582906 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a7a9081d-serving-cert\"" Nov 03 13:16:11.611281 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.611240 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1b97f-030c-43db-9438-d3d1ea58ea08-openshift-service-ca\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.611459 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.611356 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5sq\" (UniqueName: \"kubernetes.io/projected/4cd1b97f-030c-43db-9438-d3d1ea58ea08-kube-api-access-ff5sq\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.611459 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.611391 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-metrics-certs\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.611551 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.611519 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-maas-api-tls\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.713251 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.713161 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1b97f-030c-43db-9438-d3d1ea58ea08-openshift-service-ca\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.713404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.713243 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5sq\" (UniqueName: \"kubernetes.io/projected/4cd1b97f-030c-43db-9438-d3d1ea58ea08-kube-api-access-ff5sq\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.713404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.713294 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-metrics-certs\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.713404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.713364 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-maas-api-tls\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.713802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.713763 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1b97f-030c-43db-9438-d3d1ea58ea08-openshift-service-ca\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.716142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.716120 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-metrics-certs\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.716874 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.716810 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-maas-api-tls\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.785797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.772371 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5sq\" (UniqueName: \"kubernetes.io/projected/4cd1b97f-030c-43db-9438-d3d1ea58ea08-kube-api-access-ff5sq\") pod \"maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:11.785797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:11.785710 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:12.197454 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:12.197425 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh"] Nov 03 13:16:12.200261 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:16:12.200233 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd1b97f_030c_43db_9438_d3d1ea58ea08.slice/crio-2ccf9635f3145049d14e577ea3303b69f7cdee8f08a7da9e006f6b8620aa550d WatchSource:0}: Error finding container 2ccf9635f3145049d14e577ea3303b69f7cdee8f08a7da9e006f6b8620aa550d: Status 404 returned error can't find the container with id 2ccf9635f3145049d14e577ea3303b69f7cdee8f08a7da9e006f6b8620aa550d Nov 03 13:16:12.697801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:12.697748 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" event={"ID":"4cd1b97f-030c-43db-9438-d3d1ea58ea08","Type":"ContainerStarted","Data":"2ccf9635f3145049d14e577ea3303b69f7cdee8f08a7da9e006f6b8620aa550d"} Nov 03 13:16:13.703701 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:13.703664 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" event={"ID":"4cd1b97f-030c-43db-9438-d3d1ea58ea08","Type":"ContainerStarted","Data":"cfd82f6ae2885f621d30372c70040cf975229ae3a1345a5f48e31e96ab906cf7"} Nov 03 13:16:13.704101 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:13.703786 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:13.812889 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:13.812820 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" podStartSLOduration=1.902095608 podStartE2EDuration="2.812800535s" totalImagesPullingTime="910.704927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:13.812498524 +0000 UTC m=+705.126470407" watchObservedRunningTime="2026-11-03 13:16:13.812800535 +0000 UTC m=+705.126772390" Nov 03 13:16:15.569000 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:15.568950 4541 patch_prober.go:29] interesting pod/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:16:15.569436 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:15.569050 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:16:16.892455 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:16:16.892419 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:16:17.728032 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.727999 4541 generic.go:356] "Generic (PLEG): container finished" podID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerID="eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8" exitCode=0 Nov 03 13:16:17.728263 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.728076 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" event={"ID":"fdff3960-b86d-4c10-b7d7-137f9fa9f3db","Type":"ContainerDied","Data":"eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8"} Nov 03 13:16:17.728545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.728527 4541 scope.go:122] "RemoveContainer" containerID="eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8" Nov 03 13:16:17.729897 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.729863 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:16:17.730003 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.729952 4541 generic.go:356] "Generic (PLEG): container finished" podID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerID="06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b" exitCode=0 Nov 03 13:16:17.730003 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.729973 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" event={"ID":"9c077100-2f32-4ff0-9e89-99b4523c1278","Type":"ContainerDied","Data":"06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b"} Nov 03 13:16:17.730003 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.729954 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:16:17.730144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.730035 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:16:17.730348 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.730332 4541 scope.go:122] "RemoveContainer" containerID="06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b" Nov 03 13:16:17.731439 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.731406 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:16:17.731532 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.731475 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:16:17.731532 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:17.731516 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:16:18.761797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:18.760857 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" event={"ID":"9c077100-2f32-4ff0-9e89-99b4523c1278","Type":"ContainerStarted","Data":"14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7"} Nov 03 13:16:18.783009 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:18.782960 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" event={"ID":"fdff3960-b86d-4c10-b7d7-137f9fa9f3db","Type":"ContainerStarted","Data":"aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1"} Nov 03 13:16:19.642807 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:19.642133 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:16:19.642807 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:19.642570 4541 patch_prober.go:29] interesting pod/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:16:19.642807 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:19.642622 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 13:16:19.655805 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:19.647910 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:16:19.655805 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:19.648028 4541 patch_prober.go:29] interesting pod/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 13:16:19.655805 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:19.648076 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 13:16:19.717221 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:19.717195 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:16:20.653795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:20.643043 4541 patch_prober.go:29] interesting pod/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:16:20.653795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:20.643107 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 13:16:20.654851 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:20.654819 4541 patch_prober.go:29] interesting pod/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 13:16:20.655050 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:20.655023 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 13:16:21.651798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:21.647366 4541 patch_prober.go:29] interesting pod/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:16:21.651798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:21.647444 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 13:16:21.651798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:21.647869 4541 patch_prober.go:29] interesting pod/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 13:16:21.651798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:21.647933 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 13:16:22.646798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:22.642197 4541 patch_prober.go:29] interesting pod/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:16:22.646798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:22.642277 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 13:16:22.646798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:22.646093 4541 patch_prober.go:29] interesting pod/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 13:16:22.646798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:22.646153 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 13:16:23.656369 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:23.656327 4541 patch_prober.go:29] interesting pod/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:16:23.658520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:23.658068 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Nov 03 13:16:23.658520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:23.656974 4541 patch_prober.go:29] interesting pod/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 03 13:16:23.658520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:23.658468 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 03 13:16:24.642098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:24.642069 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:16:24.642341 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:24.642321 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:16:24.643081 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:24.643063 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:16:24.647974 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:24.647956 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:16:24.648223 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:24.648202 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:16:24.648884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:24.648868 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:16:46.889280 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:16:46.889244 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:16:48.568031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:48.566668 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc"] Nov 03 13:16:48.901947 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:48.901913 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl"] Nov 03 13:16:48.902234 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:48.902201 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" podUID="83fc22fa-3df0-469b-a570-72bba1b7ed38" containerName="maas-api" containerID="cri-o://b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822" gracePeriod=30 Nov 03 13:16:49.217985 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.217953 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:16:49.273327 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.273296 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fc22fa-3df0-469b-a570-72bba1b7ed38-kube-api-access-lfx7f\" (UniqueName: \"kubernetes.io/projected/83fc22fa-3df0-469b-a570-72bba1b7ed38-kube-api-access-lfx7f\") pod \"83fc22fa-3df0-469b-a570-72bba1b7ed38\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " Nov 03 13:16:49.273617 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.273357 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83fc22fa-3df0-469b-a570-72bba1b7ed38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fc22fa-3df0-469b-a570-72bba1b7ed38-openshift-service-ca\") pod \"83fc22fa-3df0-469b-a570-72bba1b7ed38\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " Nov 03 13:16:49.273617 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.273387 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-metrics-certs\") pod \"83fc22fa-3df0-469b-a570-72bba1b7ed38\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " Nov 03 13:16:49.273617 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.273432 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls\") pod \"83fc22fa-3df0-469b-a570-72bba1b7ed38\" (UID: \"83fc22fa-3df0-469b-a570-72bba1b7ed38\") " Nov 03 13:16:49.273831 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.273688 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fc22fa-3df0-469b-a570-72bba1b7ed38-openshift-service-ca" pod "83fc22fa-3df0-469b-a570-72bba1b7ed38" (UID: "83fc22fa-3df0-469b-a570-72bba1b7ed38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:49.273831 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.273815 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fc22fa-3df0-469b-a570-72bba1b7ed38-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:49.275609 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.275578 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-metrics-certs" pod "83fc22fa-3df0-469b-a570-72bba1b7ed38" (UID: "83fc22fa-3df0-469b-a570-72bba1b7ed38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:49.275719 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.275616 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls" pod "83fc22fa-3df0-469b-a570-72bba1b7ed38" (UID: "83fc22fa-3df0-469b-a570-72bba1b7ed38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:49.275719 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.275651 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fc22fa-3df0-469b-a570-72bba1b7ed38-kube-api-access-lfx7f" pod "83fc22fa-3df0-469b-a570-72bba1b7ed38" (UID: "83fc22fa-3df0-469b-a570-72bba1b7ed38"). InnerVolumeSpecName "kube-api-access-lfx7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:49.375246 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.375210 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfx7f\" (UniqueName: \"kubernetes.io/projected/83fc22fa-3df0-469b-a570-72bba1b7ed38-kube-api-access-lfx7f\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:49.375246 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.375245 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:49.375246 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.375255 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fc22fa-3df0-469b-a570-72bba1b7ed38-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:49.958796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.958614 4541 generic.go:356] "Generic (PLEG): container finished" podID="83fc22fa-3df0-469b-a570-72bba1b7ed38" containerID="b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822" exitCode=0 Nov 03 13:16:49.958796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.958669 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" event={"ID":"83fc22fa-3df0-469b-a570-72bba1b7ed38","Type":"ContainerDied","Data":"b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822"} Nov 03 13:16:49.958796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.958715 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" event={"ID":"83fc22fa-3df0-469b-a570-72bba1b7ed38","Type":"ContainerDied","Data":"af3925df52d5dde4dd30aaec2551fe3ad56753ead6d727e2c0ea2c27baf906c1"} Nov 03 13:16:49.958796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.958741 4541 scope.go:122] "RemoveContainer" containerID="b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822" Nov 03 13:16:49.959475 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.958911 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl" Nov 03 13:16:49.987704 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.987676 4541 scope.go:122] "RemoveContainer" containerID="b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822" Nov 03 13:16:49.988142 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:16:49.988119 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822\": container with ID starting with b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822 not found: ID does not exist" containerID="b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822" Nov 03 13:16:49.988220 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:49.988154 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822"} err="failed to get container status \"b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822\": rpc error: code = NotFound desc = could not find container \"b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822\": container with ID starting with b354c98fc5d1effc30c340345a45eea61614f074d34362e87ce19d230aed5822 not found: ID does not exist" Nov 03 13:16:50.123498 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:50.123463 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl"] Nov 03 13:16:50.172095 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:50.172054 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-821a13f6-568d745c84-qz8sl"] Nov 03 13:16:50.867463 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:50.867432 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83fc22fa-3df0-469b-a570-72bba1b7ed38" path="/var/lib/kubelet/pods/83fc22fa-3df0-469b-a570-72bba1b7ed38/volumes" Nov 03 13:16:53.567726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.567674 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" podUID="41040ec9-636c-47f7-a965-b9f7229deb2c" containerName="payload-pre-processing" containerID="cri-o://d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed" gracePeriod=25 Nov 03 13:16:53.877010 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.876982 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:16:53.917764 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.917724 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41040ec9-636c-47f7-a965-b9f7229deb2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41040ec9-636c-47f7-a965-b9f7229deb2c-plugins-config-volume\") pod \"41040ec9-636c-47f7-a965-b9f7229deb2c\" (UID: \"41040ec9-636c-47f7-a965-b9f7229deb2c\") " Nov 03 13:16:53.917764 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.917798 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41040ec9-636c-47f7-a965-b9f7229deb2c-kube-api-access-9gd5d\" (UniqueName: \"kubernetes.io/projected/41040ec9-636c-47f7-a965-b9f7229deb2c-kube-api-access-9gd5d\") pod \"41040ec9-636c-47f7-a965-b9f7229deb2c\" (UID: \"41040ec9-636c-47f7-a965-b9f7229deb2c\") " Nov 03 13:16:53.918260 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.918227 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41040ec9-636c-47f7-a965-b9f7229deb2c-plugins-config-volume" pod "41040ec9-636c-47f7-a965-b9f7229deb2c" (UID: "41040ec9-636c-47f7-a965-b9f7229deb2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:53.920075 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.920041 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41040ec9-636c-47f7-a965-b9f7229deb2c-kube-api-access-9gd5d" pod "41040ec9-636c-47f7-a965-b9f7229deb2c" (UID: "41040ec9-636c-47f7-a965-b9f7229deb2c"). InnerVolumeSpecName "kube-api-access-9gd5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:53.991170 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.991132 4541 generic.go:356] "Generic (PLEG): container finished" podID="41040ec9-636c-47f7-a965-b9f7229deb2c" containerID="d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed" exitCode=0 Nov 03 13:16:53.991356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.991191 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" Nov 03 13:16:53.991356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.991209 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" event={"ID":"41040ec9-636c-47f7-a965-b9f7229deb2c","Type":"ContainerDied","Data":"d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed"} Nov 03 13:16:53.991356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.991252 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc" event={"ID":"41040ec9-636c-47f7-a965-b9f7229deb2c","Type":"ContainerDied","Data":"0c9ae3ea3ecfdc1d9e5bc7c0f7a20bb578bd8f0ef60f17504b312889112b1847"} Nov 03 13:16:53.991356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:53.991275 4541 scope.go:122] "RemoveContainer" containerID="d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed" Nov 03 13:16:54.005233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:54.005207 4541 scope.go:122] "RemoveContainer" containerID="d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed" Nov 03 13:16:54.005792 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:16:54.005740 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed\": container with ID starting with d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed not found: ID does not exist" containerID="d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed" Nov 03 13:16:54.005912 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:54.005803 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed"} err="failed to get container status \"d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed\": rpc error: code = NotFound desc = could not find container \"d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed\": container with ID starting with d618234626a167773b92663a7e0e419d24d254463daf1b7f4e3058f861b301ed not found: ID does not exist" Nov 03 13:16:54.019355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:54.019329 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41040ec9-636c-47f7-a965-b9f7229deb2c-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:54.019355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:54.019354 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gd5d\" (UniqueName: \"kubernetes.io/projected/41040ec9-636c-47f7-a965-b9f7229deb2c-kube-api-access-9gd5d\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:54.160922 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:54.160886 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc"] Nov 03 13:16:54.166735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:54.166708 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-67349d1c-7c696cbfc5-l9wbc"] Nov 03 13:16:54.869171 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:54.869089 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41040ec9-636c-47f7-a965-b9f7229deb2c" path="/var/lib/kubelet/pods/41040ec9-636c-47f7-a965-b9f7229deb2c/volumes" Nov 03 13:16:55.708040 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:55.707973 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv"] Nov 03 13:16:55.708314 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:55.708211 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" podUID="70059faa-8450-4c6c-ba88-b9f452eed7ce" containerName="maas-api" containerID="cri-o://e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228" gracePeriod=30 Nov 03 13:16:55.989918 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:55.989893 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:16:56.005212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.005178 4541 generic.go:356] "Generic (PLEG): container finished" podID="70059faa-8450-4c6c-ba88-b9f452eed7ce" containerID="e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228" exitCode=0 Nov 03 13:16:56.005356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.005220 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" Nov 03 13:16:56.005356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.005265 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" event={"ID":"70059faa-8450-4c6c-ba88-b9f452eed7ce","Type":"ContainerDied","Data":"e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228"} Nov 03 13:16:56.005356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.005317 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv" event={"ID":"70059faa-8450-4c6c-ba88-b9f452eed7ce","Type":"ContainerDied","Data":"71950ecff91dc6de021c62b4a93a6bbe8235716a7fa6e378b87df90957fe70aa"} Nov 03 13:16:56.005356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.005339 4541 scope.go:122] "RemoveContainer" containerID="e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228" Nov 03 13:16:56.014103 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.014083 4541 scope.go:122] "RemoveContainer" containerID="e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228" Nov 03 13:16:56.014381 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:16:56.014362 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228\": container with ID starting with e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228 not found: ID does not exist" containerID="e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228" Nov 03 13:16:56.014446 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.014394 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228"} err="failed to get container status \"e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228\": rpc error: code = NotFound desc = could not find container \"e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228\": container with ID starting with e631194371fa3f1940de03ae589b23daa6b857e471e0088bcd7d3d2f82090228 not found: ID does not exist" Nov 03 13:16:56.051223 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.051186 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls\") pod \"70059faa-8450-4c6c-ba88-b9f452eed7ce\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " Nov 03 13:16:56.051361 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.051270 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70059faa-8450-4c6c-ba88-b9f452eed7ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70059faa-8450-4c6c-ba88-b9f452eed7ce-openshift-service-ca\") pod \"70059faa-8450-4c6c-ba88-b9f452eed7ce\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " Nov 03 13:16:56.051435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.051418 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70059faa-8450-4c6c-ba88-b9f452eed7ce-kube-api-access-wlxcn\" (UniqueName: \"kubernetes.io/projected/70059faa-8450-4c6c-ba88-b9f452eed7ce-kube-api-access-wlxcn\") pod \"70059faa-8450-4c6c-ba88-b9f452eed7ce\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " Nov 03 13:16:56.051490 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.051475 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-metrics-certs\") pod \"70059faa-8450-4c6c-ba88-b9f452eed7ce\" (UID: \"70059faa-8450-4c6c-ba88-b9f452eed7ce\") " Nov 03 13:16:56.051665 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.051631 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70059faa-8450-4c6c-ba88-b9f452eed7ce-openshift-service-ca" pod "70059faa-8450-4c6c-ba88-b9f452eed7ce" (UID: "70059faa-8450-4c6c-ba88-b9f452eed7ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:56.051889 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.051870 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70059faa-8450-4c6c-ba88-b9f452eed7ce-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.053414 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.053383 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls" pod "70059faa-8450-4c6c-ba88-b9f452eed7ce" (UID: "70059faa-8450-4c6c-ba88-b9f452eed7ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:56.053511 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.053487 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70059faa-8450-4c6c-ba88-b9f452eed7ce-kube-api-access-wlxcn" pod "70059faa-8450-4c6c-ba88-b9f452eed7ce" (UID: "70059faa-8450-4c6c-ba88-b9f452eed7ce"). InnerVolumeSpecName "kube-api-access-wlxcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:56.053595 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.053574 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-metrics-certs" pod "70059faa-8450-4c6c-ba88-b9f452eed7ce" (UID: "70059faa-8450-4c6c-ba88-b9f452eed7ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:56.152837 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.152803 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlxcn\" (UniqueName: \"kubernetes.io/projected/70059faa-8450-4c6c-ba88-b9f452eed7ce-kube-api-access-wlxcn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.152837 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.152838 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.152837 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.152851 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70059faa-8450-4c6c-ba88-b9f452eed7ce-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.420073 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.420035 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv"] Nov 03 13:16:56.425261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.425233 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-67349d1c-74746c9c-c8pxv"] Nov 03 13:16:56.867932 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:56.867900 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70059faa-8450-4c6c-ba88-b9f452eed7ce" path="/var/lib/kubelet/pods/70059faa-8450-4c6c-ba88-b9f452eed7ce/volumes" Nov 03 13:16:57.000292 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.000247 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf"] Nov 03 13:16:57.000697 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.000489 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" podUID="25c5f500-e16e-4b94-8627-0fc76293c20a" containerName="maas-api" containerID="cri-o://83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80" gracePeriod=30 Nov 03 13:16:57.250343 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.250320 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:57.363944 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.363905 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25c5f500-e16e-4b94-8627-0fc76293c20a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c5f500-e16e-4b94-8627-0fc76293c20a-openshift-service-ca\") pod \"25c5f500-e16e-4b94-8627-0fc76293c20a\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " Nov 03 13:16:57.364110 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.363993 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-metrics-certs\") pod \"25c5f500-e16e-4b94-8627-0fc76293c20a\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " Nov 03 13:16:57.364110 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.364021 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25c5f500-e16e-4b94-8627-0fc76293c20a-kube-api-access-jh6d7\" (UniqueName: \"kubernetes.io/projected/25c5f500-e16e-4b94-8627-0fc76293c20a-kube-api-access-jh6d7\") pod \"25c5f500-e16e-4b94-8627-0fc76293c20a\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " Nov 03 13:16:57.364110 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.364094 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls\") pod \"25c5f500-e16e-4b94-8627-0fc76293c20a\" (UID: \"25c5f500-e16e-4b94-8627-0fc76293c20a\") " Nov 03 13:16:57.364955 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.364923 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25c5f500-e16e-4b94-8627-0fc76293c20a-openshift-service-ca" pod "25c5f500-e16e-4b94-8627-0fc76293c20a" (UID: "25c5f500-e16e-4b94-8627-0fc76293c20a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:57.366969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.366921 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c5f500-e16e-4b94-8627-0fc76293c20a-kube-api-access-jh6d7" pod "25c5f500-e16e-4b94-8627-0fc76293c20a" (UID: "25c5f500-e16e-4b94-8627-0fc76293c20a"). InnerVolumeSpecName "kube-api-access-jh6d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:57.367169 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.367135 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-metrics-certs" pod "25c5f500-e16e-4b94-8627-0fc76293c20a" (UID: "25c5f500-e16e-4b94-8627-0fc76293c20a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:57.368390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.368362 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls" pod "25c5f500-e16e-4b94-8627-0fc76293c20a" (UID: "25c5f500-e16e-4b94-8627-0fc76293c20a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:57.465793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.465677 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25c5f500-e16e-4b94-8627-0fc76293c20a-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:57.465793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.465730 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:57.465793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.465746 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh6d7\" (UniqueName: \"kubernetes.io/projected/25c5f500-e16e-4b94-8627-0fc76293c20a-kube-api-access-jh6d7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:57.465793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:57.465760 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25c5f500-e16e-4b94-8627-0fc76293c20a-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:58.018098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.018071 4541 generic.go:356] "Generic (PLEG): container finished" podID="25c5f500-e16e-4b94-8627-0fc76293c20a" containerID="83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80" exitCode=0 Nov 03 13:16:58.018540 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.018115 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" Nov 03 13:16:58.018540 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.018162 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" event={"ID":"25c5f500-e16e-4b94-8627-0fc76293c20a","Type":"ContainerDied","Data":"83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80"} Nov 03 13:16:58.018540 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.018236 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf" event={"ID":"25c5f500-e16e-4b94-8627-0fc76293c20a","Type":"ContainerDied","Data":"f1f2a891d6c2bca537f3439feab1ad5f8804467667b5aa66d77b0309a8dd43e0"} Nov 03 13:16:58.018540 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.018269 4541 scope.go:122] "RemoveContainer" containerID="83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80" Nov 03 13:16:58.019900 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.019876 4541 generic.go:356] "Generic (PLEG): container finished" podID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerID="8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa" exitCode=0 Nov 03 13:16:58.019994 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.019957 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" event={"ID":"7bd25c3f-f82a-4d42-aadb-29f6398f44f1","Type":"ContainerDied","Data":"8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa"} Nov 03 13:16:58.020445 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.020427 4541 scope.go:122] "RemoveContainer" containerID="8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa" Nov 03 13:16:58.021703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.021674 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:16:58.021763 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.021749 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:16:58.021824 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.021800 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:16:58.028288 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.028133 4541 scope.go:122] "RemoveContainer" containerID="83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80" Nov 03 13:16:58.028538 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:16:58.028519 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80\": container with ID starting with 83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80 not found: ID does not exist" containerID="83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80" Nov 03 13:16:58.028609 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.028549 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80"} err="failed to get container status \"83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80\": rpc error: code = NotFound desc = could not find container \"83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80\": container with ID starting with 83efb41d7a47e44cf71493db199a7ff23c2aa66c3f4868e06428929170ef6c80 not found: ID does not exist" Nov 03 13:16:58.173530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.173495 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf"] Nov 03 13:16:58.178953 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.178920 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cb1bd0b2-7f6644ffc9-zk2lf"] Nov 03 13:16:58.867358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:58.867328 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25c5f500-e16e-4b94-8627-0fc76293c20a" path="/var/lib/kubelet/pods/25c5f500-e16e-4b94-8627-0fc76293c20a/volumes" Nov 03 13:16:59.031370 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:59.031335 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" event={"ID":"7bd25c3f-f82a-4d42-aadb-29f6398f44f1","Type":"ContainerStarted","Data":"9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834"} Nov 03 13:16:59.390944 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:59.390907 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh"] Nov 03 13:16:59.391154 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:59.391126 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" containerID="cri-o://aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1" gracePeriod=30 Nov 03 13:16:59.458798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:59.456953 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:16:59.458798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:59.457112 4541 patch_prober.go:29] interesting pod/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:16:59.458798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:59.457159 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 13:16:59.599349 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:59.599316 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4"] Nov 03 13:16:59.600001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:16:59.599968 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" containerID="cri-o://14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7" gracePeriod=30 Nov 03 13:17:00.457743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:00.457248 4541 patch_prober.go:29] interesting pod/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:17:00.457743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:00.457326 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 13:17:01.457297 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:01.457254 4541 patch_prober.go:29] interesting pod/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:17:01.457591 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:01.457561 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 13:17:02.459257 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:02.459211 4541 patch_prober.go:29] interesting pod/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:17:02.459806 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:02.459280 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 13:17:03.458796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.458248 4541 patch_prober.go:29] interesting pod/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:17:03.458796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.458315 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 03 13:17:03.833308 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833227 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws"] Nov 03 13:17:03.833674 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833642 4541 cpu_manager.go:414] "containerMap: removing container" podUID="25c5f500-e16e-4b94-8627-0fc76293c20a" containerName="maas-api" Nov 03 13:17:03.833674 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833653 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25c5f500-e16e-4b94-8627-0fc76293c20a" containerName="maas-api" Nov 03 13:17:03.833674 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833668 4541 cpu_manager.go:414] "containerMap: removing container" podUID="83fc22fa-3df0-469b-a570-72bba1b7ed38" containerName="maas-api" Nov 03 13:17:03.833674 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833673 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fc22fa-3df0-469b-a570-72bba1b7ed38" containerName="maas-api" Nov 03 13:17:03.833843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833695 4541 cpu_manager.go:414] "containerMap: removing container" podUID="41040ec9-636c-47f7-a965-b9f7229deb2c" containerName="payload-pre-processing" Nov 03 13:17:03.833843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833701 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41040ec9-636c-47f7-a965-b9f7229deb2c" containerName="payload-pre-processing" Nov 03 13:17:03.833843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833708 4541 cpu_manager.go:414] "containerMap: removing container" podUID="70059faa-8450-4c6c-ba88-b9f452eed7ce" containerName="maas-api" Nov 03 13:17:03.833843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833713 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70059faa-8450-4c6c-ba88-b9f452eed7ce" containerName="maas-api" Nov 03 13:17:03.833843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833795 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="41040ec9-636c-47f7-a965-b9f7229deb2c" containerName="payload-pre-processing" Nov 03 13:17:03.833843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833806 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="70059faa-8450-4c6c-ba88-b9f452eed7ce" containerName="maas-api" Nov 03 13:17:03.833843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833817 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="25c5f500-e16e-4b94-8627-0fc76293c20a" containerName="maas-api" Nov 03 13:17:03.833843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.833828 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fc22fa-3df0-469b-a570-72bba1b7ed38" containerName="maas-api" Nov 03 13:17:03.855169 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.855137 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:03.880066 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.880026 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh"] Nov 03 13:17:03.880291 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.880260 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" podUID="4cd1b97f-030c-43db-9438-d3d1ea58ea08" containerName="maas-api" containerID="cri-o://cfd82f6ae2885f621d30372c70040cf975229ae3a1345a5f48e31e96ab906cf7" gracePeriod=30 Nov 03 13:17:03.891271 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.891234 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws"] Nov 03 13:17:03.905346 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.905319 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-cdb532-dockercfg-pdl6v\"" Nov 03 13:17:03.905464 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.905348 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-cdb532\"" Nov 03 13:17:03.929895 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.929855 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkw2c\" (UniqueName: \"kubernetes.io/projected/db992848-af0b-4a8d-aa98-711ff341ea78-kube-api-access-tkw2c\") pod \"payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws\" (UID: \"db992848-af0b-4a8d-aa98-711ff341ea78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:03.930063 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.929925 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db992848-af0b-4a8d-aa98-711ff341ea78-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws\" (UID: \"db992848-af0b-4a8d-aa98-711ff341ea78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:03.985886 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.985853 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t"] Nov 03 13:17:03.988442 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:03.988415 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:04.030579 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.030525 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lql4g\" (UniqueName: \"kubernetes.io/projected/3b4e8e41-848f-4bb0-a195-d3a1282908b2-kube-api-access-lql4g\") pod \"payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t\" (UID: \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:04.030579 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.030584 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4e8e41-848f-4bb0-a195-d3a1282908b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t\" (UID: \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:04.030872 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.030745 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkw2c\" (UniqueName: \"kubernetes.io/projected/db992848-af0b-4a8d-aa98-711ff341ea78-kube-api-access-tkw2c\") pod \"payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws\" (UID: \"db992848-af0b-4a8d-aa98-711ff341ea78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:04.030872 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.030826 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db992848-af0b-4a8d-aa98-711ff341ea78-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws\" (UID: \"db992848-af0b-4a8d-aa98-711ff341ea78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:04.031612 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.031586 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db992848-af0b-4a8d-aa98-711ff341ea78-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws\" (UID: \"db992848-af0b-4a8d-aa98-711ff341ea78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:04.047435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.047402 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t"] Nov 03 13:17:04.112042 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.111821 4541 generic.go:356] "Generic (PLEG): container finished" podID="4cd1b97f-030c-43db-9438-d3d1ea58ea08" containerID="cfd82f6ae2885f621d30372c70040cf975229ae3a1345a5f48e31e96ab906cf7" exitCode=0 Nov 03 13:17:04.112042 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.111886 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" event={"ID":"4cd1b97f-030c-43db-9438-d3d1ea58ea08","Type":"ContainerDied","Data":"cfd82f6ae2885f621d30372c70040cf975229ae3a1345a5f48e31e96ab906cf7"} Nov 03 13:17:04.120349 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.120318 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkw2c\" (UniqueName: \"kubernetes.io/projected/db992848-af0b-4a8d-aa98-711ff341ea78-kube-api-access-tkw2c\") pod \"payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws\" (UID: \"db992848-af0b-4a8d-aa98-711ff341ea78\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:04.132037 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.131962 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lql4g\" (UniqueName: \"kubernetes.io/projected/3b4e8e41-848f-4bb0-a195-d3a1282908b2-kube-api-access-lql4g\") pod \"payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t\" (UID: \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:04.132224 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.132043 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4e8e41-848f-4bb0-a195-d3a1282908b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t\" (UID: \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:04.133531 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.133500 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4e8e41-848f-4bb0-a195-d3a1282908b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t\" (UID: \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:04.170499 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.170468 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:04.187015 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.186988 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:17:04.192328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.192296 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lql4g\" (UniqueName: \"kubernetes.io/projected/3b4e8e41-848f-4bb0-a195-d3a1282908b2-kube-api-access-lql4g\") pod \"payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t\" (UID: \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:04.233144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.232747 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-metrics-certs\") pod \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " Nov 03 13:17:04.233144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.232840 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd1b97f-030c-43db-9438-d3d1ea58ea08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1b97f-030c-43db-9438-d3d1ea58ea08-openshift-service-ca\") pod \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " Nov 03 13:17:04.233144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.232878 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-maas-api-tls\") pod \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " Nov 03 13:17:04.233962 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.233455 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd1b97f-030c-43db-9438-d3d1ea58ea08-kube-api-access-ff5sq\" (UniqueName: \"kubernetes.io/projected/4cd1b97f-030c-43db-9438-d3d1ea58ea08-kube-api-access-ff5sq\") pod \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\" (UID: \"4cd1b97f-030c-43db-9438-d3d1ea58ea08\") " Nov 03 13:17:04.234480 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.234151 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd1b97f-030c-43db-9438-d3d1ea58ea08-openshift-service-ca" pod "4cd1b97f-030c-43db-9438-d3d1ea58ea08" (UID: "4cd1b97f-030c-43db-9438-d3d1ea58ea08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:04.237678 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.237649 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-maas-api-tls" pod "4cd1b97f-030c-43db-9438-d3d1ea58ea08" (UID: "4cd1b97f-030c-43db-9438-d3d1ea58ea08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:17:04.242909 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.239728 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd1b97f-030c-43db-9438-d3d1ea58ea08-kube-api-access-ff5sq" pod "4cd1b97f-030c-43db-9438-d3d1ea58ea08" (UID: "4cd1b97f-030c-43db-9438-d3d1ea58ea08"). InnerVolumeSpecName "kube-api-access-ff5sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:04.242909 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.239917 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-metrics-certs" pod "4cd1b97f-030c-43db-9438-d3d1ea58ea08" (UID: "4cd1b97f-030c-43db-9438-d3d1ea58ea08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:17:04.301575 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.301526 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:04.335368 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.335333 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd1b97f-030c-43db-9438-d3d1ea58ea08-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.335368 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.335365 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.335368 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.335377 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff5sq\" (UniqueName: \"kubernetes.io/projected/4cd1b97f-030c-43db-9438-d3d1ea58ea08-kube-api-access-ff5sq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.335606 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.335386 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cd1b97f-030c-43db-9438-d3d1ea58ea08-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.462192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.462161 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:17:04.462507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.462465 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:17:04.463153 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.463131 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:17:04.556690 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.556664 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws"] Nov 03 13:17:04.558422 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:04.558390 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb992848_af0b_4a8d_aa98_711ff341ea78.slice/crio-28dbf344b4dc76a95f593b11018a4f1a5fea9c3a9c76b6671ff3df98c2699ed9 WatchSource:0}: Error finding container 28dbf344b4dc76a95f593b11018a4f1a5fea9c3a9c76b6671ff3df98c2699ed9: Status 404 returned error can't find the container with id 28dbf344b4dc76a95f593b11018a4f1a5fea9c3a9c76b6671ff3df98c2699ed9 Nov 03 13:17:04.678509 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.678483 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t"] Nov 03 13:17:04.681941 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:04.681912 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4e8e41_848f_4bb0_a195_d3a1282908b2.slice/crio-fc62bce7e8b8a37b55cdfc1edb74181cfbde9b45275c45b73483344eae363274 WatchSource:0}: Error finding container fc62bce7e8b8a37b55cdfc1edb74181cfbde9b45275c45b73483344eae363274: Status 404 returned error can't find the container with id fc62bce7e8b8a37b55cdfc1edb74181cfbde9b45275c45b73483344eae363274 Nov 03 13:17:04.718443 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.718373 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:17:04.848492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.848456 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-podinfo\") pod \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " Nov 03 13:17:04.848961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.848507 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-token\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-token\") pod \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " Nov 03 13:17:04.848961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.848554 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-credential-socket\") pod \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " Nov 03 13:17:04.848961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.848655 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-certs\") pod \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " Nov 03 13:17:04.848961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.848699 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-data\") pod \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " Nov 03 13:17:04.848961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.848727 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istiod-ca-cert\") pod \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " Nov 03 13:17:04.848961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.848792 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-socket\") pod \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " Nov 03 13:17:04.848961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.848850 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-kube-api-access-26hgw\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-kube-api-access-26hgw\") pod \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " Nov 03 13:17:04.848961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.848894 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-envoy\") pod \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\" (UID: \"fdff3960-b86d-4c10-b7d7-137f9fa9f3db\") " Nov 03 13:17:04.849393 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.849141 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-data" pod "fdff3960-b86d-4c10-b7d7-137f9fa9f3db" (UID: "fdff3960-b86d-4c10-b7d7-137f9fa9f3db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.849393 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.849285 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-certs" pod "fdff3960-b86d-4c10-b7d7-137f9fa9f3db" (UID: "fdff3960-b86d-4c10-b7d7-137f9fa9f3db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.849393 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.849315 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-credential-socket" pod "fdff3960-b86d-4c10-b7d7-137f9fa9f3db" (UID: "fdff3960-b86d-4c10-b7d7-137f9fa9f3db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.849560 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.849416 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-socket" pod "fdff3960-b86d-4c10-b7d7-137f9fa9f3db" (UID: "fdff3960-b86d-4c10-b7d7-137f9fa9f3db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.849560 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.849474 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.849560 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.849493 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.849560 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.849507 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.849790 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.849624 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istiod-ca-cert" pod "fdff3960-b86d-4c10-b7d7-137f9fa9f3db" (UID: "fdff3960-b86d-4c10-b7d7-137f9fa9f3db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:04.852743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.852713 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-token" pod "fdff3960-b86d-4c10-b7d7-137f9fa9f3db" (UID: "fdff3960-b86d-4c10-b7d7-137f9fa9f3db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:04.852898 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.852743 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-envoy" pod "fdff3960-b86d-4c10-b7d7-137f9fa9f3db" (UID: "fdff3960-b86d-4c10-b7d7-137f9fa9f3db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.852898 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.852805 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-kube-api-access-26hgw" pod "fdff3960-b86d-4c10-b7d7-137f9fa9f3db" (UID: "fdff3960-b86d-4c10-b7d7-137f9fa9f3db"). InnerVolumeSpecName "kube-api-access-26hgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:04.853021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.852922 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-podinfo" pod "fdff3960-b86d-4c10-b7d7-137f9fa9f3db" (UID: "fdff3960-b86d-4c10-b7d7-137f9fa9f3db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:17:04.873532 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.873505 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:17:04.950239 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950164 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-podinfo\") pod \"9c077100-2f32-4ff0-9e89-99b4523c1278\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " Nov 03 13:17:04.950239 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950224 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-credential-socket\") pod \"9c077100-2f32-4ff0-9e89-99b4523c1278\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " Nov 03 13:17:04.950424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950296 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-envoy\") pod \"9c077100-2f32-4ff0-9e89-99b4523c1278\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " Nov 03 13:17:04.950424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950358 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-data\") pod \"9c077100-2f32-4ff0-9e89-99b4523c1278\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " Nov 03 13:17:04.950424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950385 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-certs\") pod \"9c077100-2f32-4ff0-9e89-99b4523c1278\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " Nov 03 13:17:04.950424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950415 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c077100-2f32-4ff0-9e89-99b4523c1278-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c077100-2f32-4ff0-9e89-99b4523c1278-istiod-ca-cert\") pod \"9c077100-2f32-4ff0-9e89-99b4523c1278\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " Nov 03 13:17:04.950633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950441 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-token\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-token\") pod \"9c077100-2f32-4ff0-9e89-99b4523c1278\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " Nov 03 13:17:04.950633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950466 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-credential-socket" pod "9c077100-2f32-4ff0-9e89-99b4523c1278" (UID: "9c077100-2f32-4ff0-9e89-99b4523c1278"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.950633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950484 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-kube-api-access-p7hzh\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-kube-api-access-p7hzh\") pod \"9c077100-2f32-4ff0-9e89-99b4523c1278\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " Nov 03 13:17:04.950633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950574 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-socket\") pod \"9c077100-2f32-4ff0-9e89-99b4523c1278\" (UID: \"9c077100-2f32-4ff0-9e89-99b4523c1278\") " Nov 03 13:17:04.950633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950624 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-data" pod "9c077100-2f32-4ff0-9e89-99b4523c1278" (UID: "9c077100-2f32-4ff0-9e89-99b4523c1278"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.950885 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950673 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-certs" pod "9c077100-2f32-4ff0-9e89-99b4523c1278" (UID: "9c077100-2f32-4ff0-9e89-99b4523c1278"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.950959 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950937 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-socket" pod "9c077100-2f32-4ff0-9e89-99b4523c1278" (UID: "9c077100-2f32-4ff0-9e89-99b4523c1278"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.951031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950980 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c077100-2f32-4ff0-9e89-99b4523c1278-istiod-ca-cert" pod "9c077100-2f32-4ff0-9e89-99b4523c1278" (UID: "9c077100-2f32-4ff0-9e89-99b4523c1278"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:04.951031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.950999 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.951139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.951039 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hgw\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-kube-api-access-26hgw\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.951139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.951057 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.951139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.951072 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.951139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.951087 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.951139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.951101 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.951139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.951115 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.951139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.951129 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.951485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.951145 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdff3960-b86d-4c10-b7d7-137f9fa9f3db-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:04.952402 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.952365 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-podinfo" pod "9c077100-2f32-4ff0-9e89-99b4523c1278" (UID: "9c077100-2f32-4ff0-9e89-99b4523c1278"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:17:04.952695 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.952677 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-envoy" pod "9c077100-2f32-4ff0-9e89-99b4523c1278" (UID: "9c077100-2f32-4ff0-9e89-99b4523c1278"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:04.952861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.952844 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-kube-api-access-p7hzh" pod "9c077100-2f32-4ff0-9e89-99b4523c1278" (UID: "9c077100-2f32-4ff0-9e89-99b4523c1278"). InnerVolumeSpecName "kube-api-access-p7hzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:04.952908 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:04.952850 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-token" pod "9c077100-2f32-4ff0-9e89-99b4523c1278" (UID: "9c077100-2f32-4ff0-9e89-99b4523c1278"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:05.051844 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.051798 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c077100-2f32-4ff0-9e89-99b4523c1278-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:05.051844 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.051832 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:05.051844 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.051842 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7hzh\" (UniqueName: \"kubernetes.io/projected/9c077100-2f32-4ff0-9e89-99b4523c1278-kube-api-access-p7hzh\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:05.052089 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.051853 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:05.052089 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.051867 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:05.052089 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.051876 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c077100-2f32-4ff0-9e89-99b4523c1278-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:05.118838 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.118805 4541 generic.go:356] "Generic (PLEG): container finished" podID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerID="14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7" exitCode=0 Nov 03 13:17:05.119021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.118891 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" event={"ID":"9c077100-2f32-4ff0-9e89-99b4523c1278","Type":"ContainerDied","Data":"14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7"} Nov 03 13:17:05.119021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.118908 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" Nov 03 13:17:05.119021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.118940 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4" event={"ID":"9c077100-2f32-4ff0-9e89-99b4523c1278","Type":"ContainerDied","Data":"402efebc258f0b85623cf883fe53567b2cc7d0d16370465abb30da8a645661c6"} Nov 03 13:17:05.119021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.118964 4541 scope.go:122] "RemoveContainer" containerID="14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7" Nov 03 13:17:05.120709 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.120689 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" Nov 03 13:17:05.120910 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.120688 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh" event={"ID":"4cd1b97f-030c-43db-9438-d3d1ea58ea08","Type":"ContainerDied","Data":"2ccf9635f3145049d14e577ea3303b69f7cdee8f08a7da9e006f6b8620aa550d"} Nov 03 13:17:05.122431 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.122410 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" event={"ID":"3b4e8e41-848f-4bb0-a195-d3a1282908b2","Type":"ContainerStarted","Data":"e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed"} Nov 03 13:17:05.122543 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.122453 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" event={"ID":"3b4e8e41-848f-4bb0-a195-d3a1282908b2","Type":"ContainerStarted","Data":"fc62bce7e8b8a37b55cdfc1edb74181cfbde9b45275c45b73483344eae363274"} Nov 03 13:17:05.123799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.122919 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:05.124788 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.124750 4541 generic.go:356] "Generic (PLEG): container finished" podID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerID="aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1" exitCode=0 Nov 03 13:17:05.124901 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.124807 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" Nov 03 13:17:05.124901 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.124816 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" event={"ID":"fdff3960-b86d-4c10-b7d7-137f9fa9f3db","Type":"ContainerDied","Data":"aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1"} Nov 03 13:17:05.124901 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.124868 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh" event={"ID":"fdff3960-b86d-4c10-b7d7-137f9fa9f3db","Type":"ContainerDied","Data":"683396a4a0119bc594674e502fc7d12da2ea939c20b8832cc18d4083b234c56b"} Nov 03 13:17:05.126899 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.126808 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" event={"ID":"db992848-af0b-4a8d-aa98-711ff341ea78","Type":"ContainerStarted","Data":"6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600"} Nov 03 13:17:05.126899 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.126866 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" event={"ID":"db992848-af0b-4a8d-aa98-711ff341ea78","Type":"ContainerStarted","Data":"28dbf344b4dc76a95f593b11018a4f1a5fea9c3a9c76b6671ff3df98c2699ed9"} Nov 03 13:17:05.130799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.130712 4541 scope.go:122] "RemoveContainer" containerID="06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b" Nov 03 13:17:05.139223 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.139206 4541 scope.go:122] "RemoveContainer" containerID="14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7" Nov 03 13:17:05.139491 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:05.139472 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7\": container with ID starting with 14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7 not found: ID does not exist" containerID="14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7" Nov 03 13:17:05.139534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.139500 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7"} err="failed to get container status \"14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7\": rpc error: code = NotFound desc = could not find container \"14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7\": container with ID starting with 14eee87f49c08498e9b918d36378523d91b7ee7e7fc482271acde10f7dc695f7 not found: ID does not exist" Nov 03 13:17:05.139534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.139518 4541 scope.go:122] "RemoveContainer" containerID="06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b" Nov 03 13:17:05.139759 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:05.139743 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b\": container with ID starting with 06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b not found: ID does not exist" containerID="06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b" Nov 03 13:17:05.139865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.139780 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b"} err="failed to get container status \"06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b\": rpc error: code = NotFound desc = could not find container \"06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b\": container with ID starting with 06e06bafcd6c25d53f0a087239e3d0b8ef47bdb3c2164541462262557420991b not found: ID does not exist" Nov 03 13:17:05.139865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.139793 4541 scope.go:122] "RemoveContainer" containerID="cfd82f6ae2885f621d30372c70040cf975229ae3a1345a5f48e31e96ab906cf7" Nov 03 13:17:05.147740 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.147723 4541 scope.go:122] "RemoveContainer" containerID="aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1" Nov 03 13:17:05.155900 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.155877 4541 scope.go:122] "RemoveContainer" containerID="eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8" Nov 03 13:17:05.163574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.163549 4541 scope.go:122] "RemoveContainer" containerID="aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1" Nov 03 13:17:05.163884 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:05.163852 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1\": container with ID starting with aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1 not found: ID does not exist" containerID="aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1" Nov 03 13:17:05.163974 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.163891 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1"} err="failed to get container status \"aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1\": rpc error: code = NotFound desc = could not find container \"aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1\": container with ID starting with aac2dbc5e4f3c5e3360d7c0d26ae373cc043da5a5d1853188f6d6986541b90e1 not found: ID does not exist" Nov 03 13:17:05.163974 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.163910 4541 scope.go:122] "RemoveContainer" containerID="eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8" Nov 03 13:17:05.164172 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:05.164150 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8\": container with ID starting with eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8 not found: ID does not exist" containerID="eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8" Nov 03 13:17:05.164230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.164182 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8"} err="failed to get container status \"eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8\": rpc error: code = NotFound desc = could not find container \"eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8\": container with ID starting with eaf263d84c225bed5e10257442ec8a076072da487ea76d8dbf43de3d0109b1a8 not found: ID does not exist" Nov 03 13:17:05.274433 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.274354 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4"] Nov 03 13:17:05.277763 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.277735 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-67349d1c-openshift-default-6495c4f4cf-xndk4"] Nov 03 13:17:05.382141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.382094 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" podStartSLOduration=2.382076546 podStartE2EDuration="2.382076546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:05.381793667 +0000 UTC m=+756.695765519" watchObservedRunningTime="2026-11-03 13:17:05.382076546 +0000 UTC m=+756.696048406" Nov 03 13:17:05.486347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.486311 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh"] Nov 03 13:17:05.492765 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.492735 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a7a9081d-5b4dd6575b-pjtgh"] Nov 03 13:17:05.598753 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.598651 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" podStartSLOduration=2.5986386980000002 podStartE2EDuration="2.598638698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:05.597038082 +0000 UTC m=+756.911009943" watchObservedRunningTime="2026-11-03 13:17:05.598638698 +0000 UTC m=+756.912610555" Nov 03 13:17:05.702287 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.702249 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh"] Nov 03 13:17:05.709240 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:05.709211 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-cb1bd0b2-gw-openshift-default-76cc96cf49-4jdgh"] Nov 03 13:17:06.135088 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:06.135058 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:06.867614 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:06.867577 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd1b97f-030c-43db-9438-d3d1ea58ea08" path="/var/lib/kubelet/pods/4cd1b97f-030c-43db-9438-d3d1ea58ea08/volumes" Nov 03 13:17:06.868147 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:06.868129 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" path="/var/lib/kubelet/pods/9c077100-2f32-4ff0-9e89-99b4523c1278/volumes" Nov 03 13:17:06.868636 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:06.868618 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" path="/var/lib/kubelet/pods/fdff3960-b86d-4c10-b7d7-137f9fa9f3db/volumes" Nov 03 13:17:11.314957 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.314873 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s"] Nov 03 13:17:11.315347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315269 4541 cpu_manager.go:414] "containerMap: removing container" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" Nov 03 13:17:11.315347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315280 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" Nov 03 13:17:11.315347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315303 4541 cpu_manager.go:414] "containerMap: removing container" podUID="4cd1b97f-030c-43db-9438-d3d1ea58ea08" containerName="maas-api" Nov 03 13:17:11.315347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315308 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd1b97f-030c-43db-9438-d3d1ea58ea08" containerName="maas-api" Nov 03 13:17:11.315347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315318 4541 cpu_manager.go:414] "containerMap: removing container" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" Nov 03 13:17:11.315347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315322 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" Nov 03 13:17:11.315347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315329 4541 cpu_manager.go:414] "containerMap: removing container" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" Nov 03 13:17:11.315347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315334 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" Nov 03 13:17:11.315607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315393 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd1b97f-030c-43db-9438-d3d1ea58ea08" containerName="maas-api" Nov 03 13:17:11.315607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315402 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" Nov 03 13:17:11.315607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315408 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdff3960-b86d-4c10-b7d7-137f9fa9f3db" containerName="istio-proxy" Nov 03 13:17:11.315607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315413 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" Nov 03 13:17:11.315607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.315420 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" Nov 03 13:17:11.318388 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.318368 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.367876 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.367838 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-cdb532-serving-cert\"" Nov 03 13:17:11.408721 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.408683 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1045d728-1c5d-49bf-9459-633d5d11aee3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.408906 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.408845 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.408906 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.408883 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47m8v\" (UniqueName: \"kubernetes.io/projected/1045d728-1c5d-49bf-9459-633d5d11aee3-kube-api-access-47m8v\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.408982 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.408957 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-metrics-certs\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.416682 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.416655 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s"] Nov 03 13:17:11.509828 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.509790 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.509998 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.509841 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47m8v\" (UniqueName: \"kubernetes.io/projected/1045d728-1c5d-49bf-9459-633d5d11aee3-kube-api-access-47m8v\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.509998 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:11.509886 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-serving-cert: secret "maas-api-e2e-worker-w2-cdb532-serving-cert" not found Nov 03 13:17:11.509998 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.509915 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-metrics-certs\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.509998 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:11.509955 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls podName:1045d728-1c5d-49bf-9459-633d5d11aee3 nodeName:}" failed. No retries permitted until 2026-11-03 13:17:12.0099336 +0000 UTC m=+763.323905446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls") pod "maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" (UID: "1045d728-1c5d-49bf-9459-633d5d11aee3") : secret "maas-api-e2e-worker-w2-cdb532-serving-cert" not found Nov 03 13:17:11.509998 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.509987 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1045d728-1c5d-49bf-9459-633d5d11aee3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.510676 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.510653 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1045d728-1c5d-49bf-9459-633d5d11aee3-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.512495 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.512474 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-metrics-certs\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:11.564339 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:11.564309 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47m8v\" (UniqueName: \"kubernetes.io/projected/1045d728-1c5d-49bf-9459-633d5d11aee3-kube-api-access-47m8v\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:12.015205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.015151 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:12.017694 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.017667 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls\") pod \"maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:12.229388 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.229351 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:12.461914 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.461879 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh"] Nov 03 13:17:12.462312 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.462299 4541 cpu_manager.go:414] "containerMap: removing container" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" Nov 03 13:17:12.462352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.462313 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c077100-2f32-4ff0-9e89-99b4523c1278" containerName="istio-proxy" Nov 03 13:17:12.464821 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.464766 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:12.514183 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.514153 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3c5dbc\"" Nov 03 13:17:12.514183 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.514164 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3c5dbc-dockercfg-m26zh\"" Nov 03 13:17:12.519031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.519004 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh"] Nov 03 13:17:12.521470 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.521444 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bfbj\" (UniqueName: \"kubernetes.io/projected/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-kube-api-access-2bfbj\") pod \"payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh\" (UID: \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:12.521610 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.521562 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh\" (UID: \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:12.601162 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.601076 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p"] Nov 03 13:17:12.603377 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.603356 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:12.622389 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.622363 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfbj\" (UniqueName: \"kubernetes.io/projected/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-kube-api-access-2bfbj\") pod \"payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh\" (UID: \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:12.622517 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.622418 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh\" (UID: \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:12.623114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.623092 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh\" (UID: \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:12.631271 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.631251 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s"] Nov 03 13:17:12.634027 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:12.633247 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1045d728_1c5d_49bf_9459_633d5d11aee3.slice/crio-2e0cc89427b01253b55b2ebef42b7d3012a622740ec78a2cd6b22eac3d977478 WatchSource:0}: Error finding container 2e0cc89427b01253b55b2ebef42b7d3012a622740ec78a2cd6b22eac3d977478: Status 404 returned error can't find the container with id 2e0cc89427b01253b55b2ebef42b7d3012a622740ec78a2cd6b22eac3d977478 Nov 03 13:17:12.667245 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.667215 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p"] Nov 03 13:17:12.684874 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.684842 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfbj\" (UniqueName: \"kubernetes.io/projected/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-kube-api-access-2bfbj\") pod \"payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh\" (UID: \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:12.723465 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.723428 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtlmq\" (UniqueName: \"kubernetes.io/projected/fc73c052-3aed-4795-81a4-edfcc8ef1033-kube-api-access-vtlmq\") pod \"payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p\" (UID: \"fc73c052-3aed-4795-81a4-edfcc8ef1033\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:12.723627 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.723489 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73c052-3aed-4795-81a4-edfcc8ef1033-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p\" (UID: \"fc73c052-3aed-4795-81a4-edfcc8ef1033\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:12.777155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.777117 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:12.825160 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.825121 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtlmq\" (UniqueName: \"kubernetes.io/projected/fc73c052-3aed-4795-81a4-edfcc8ef1033-kube-api-access-vtlmq\") pod \"payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p\" (UID: \"fc73c052-3aed-4795-81a4-edfcc8ef1033\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:12.825268 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.825208 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73c052-3aed-4795-81a4-edfcc8ef1033-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p\" (UID: \"fc73c052-3aed-4795-81a4-edfcc8ef1033\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:12.826238 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.826207 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73c052-3aed-4795-81a4-edfcc8ef1033-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p\" (UID: \"fc73c052-3aed-4795-81a4-edfcc8ef1033\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:12.879225 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.879158 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtlmq\" (UniqueName: \"kubernetes.io/projected/fc73c052-3aed-4795-81a4-edfcc8ef1033-kube-api-access-vtlmq\") pod \"payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p\" (UID: \"fc73c052-3aed-4795-81a4-edfcc8ef1033\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:12.915072 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:12.915034 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:13.138960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:13.138930 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh"] Nov 03 13:17:13.140762 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:13.140727 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1f7135_4ef0_4ac3_8bc7_ed4fa6aa89e9.slice/crio-02700fd21bf48adaa633e87cdabd58434c604c110b778932d873292a1d1fa640 WatchSource:0}: Error finding container 02700fd21bf48adaa633e87cdabd58434c604c110b778932d873292a1d1fa640: Status 404 returned error can't find the container with id 02700fd21bf48adaa633e87cdabd58434c604c110b778932d873292a1d1fa640 Nov 03 13:17:13.172252 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:13.172082 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" event={"ID":"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9","Type":"ContainerStarted","Data":"02700fd21bf48adaa633e87cdabd58434c604c110b778932d873292a1d1fa640"} Nov 03 13:17:13.173507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:13.173480 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" event={"ID":"1045d728-1c5d-49bf-9459-633d5d11aee3","Type":"ContainerStarted","Data":"2e0cc89427b01253b55b2ebef42b7d3012a622740ec78a2cd6b22eac3d977478"} Nov 03 13:17:13.280369 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:13.280346 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p"] Nov 03 13:17:13.283037 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:13.283007 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc73c052_3aed_4795_81a4_edfcc8ef1033.slice/crio-022d5b1238aec4b3bd78f4733db10875036b1a4639a3047c6ab4ac261754d4f1 WatchSource:0}: Error finding container 022d5b1238aec4b3bd78f4733db10875036b1a4639a3047c6ab4ac261754d4f1: Status 404 returned error can't find the container with id 022d5b1238aec4b3bd78f4733db10875036b1a4639a3047c6ab4ac261754d4f1 Nov 03 13:17:14.153630 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.153596 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7"] Nov 03 13:17:14.154143 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.153895 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" containerID="cri-o://9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834" gracePeriod=30 Nov 03 13:17:14.179920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.179887 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" event={"ID":"fc73c052-3aed-4795-81a4-edfcc8ef1033","Type":"ContainerStarted","Data":"3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353"} Nov 03 13:17:14.180116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.179941 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" event={"ID":"fc73c052-3aed-4795-81a4-edfcc8ef1033","Type":"ContainerStarted","Data":"022d5b1238aec4b3bd78f4733db10875036b1a4639a3047c6ab4ac261754d4f1"} Nov 03 13:17:14.180116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.179970 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:14.181532 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.181500 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" event={"ID":"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9","Type":"ContainerStarted","Data":"08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3"} Nov 03 13:17:14.181674 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.181558 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:14.183175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.183149 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" event={"ID":"1045d728-1c5d-49bf-9459-633d5d11aee3","Type":"ContainerStarted","Data":"be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677"} Nov 03 13:17:14.183288 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.183227 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:14.371802 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.371726 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" podStartSLOduration=2.371709266 podStartE2EDuration="2.371709266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:14.370836617 +0000 UTC m=+765.684808477" watchObservedRunningTime="2026-11-03 13:17:14.371709266 +0000 UTC m=+765.685681126" Nov 03 13:17:14.482854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.482738 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" podStartSLOduration=2.524526243 podStartE2EDuration="3.482723098s" totalImagesPullingTime="958.196855ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:14.4806975 +0000 UTC m=+765.794669360" watchObservedRunningTime="2026-11-03 13:17:14.482723098 +0000 UTC m=+765.796694957" Nov 03 13:17:14.591357 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.591174 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" podStartSLOduration=2.591157376 podStartE2EDuration="2.591157376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:14.590430971 +0000 UTC m=+765.904402833" watchObservedRunningTime="2026-11-03 13:17:14.591157376 +0000 UTC m=+765.905129236" Nov 03 13:17:14.997558 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:14.997519 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq"] Nov 03 13:17:15.000097 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.000077 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:15.049963 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.049933 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-5bb463\"" Nov 03 13:17:15.049963 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.049931 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-5bb463-dockercfg-lfrxh\"" Nov 03 13:17:15.056831 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.056801 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq"] Nov 03 13:17:15.145838 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.145715 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e528f67e-a234-452d-8c81-3e8f2f0d31ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq\" (UID: \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:15.145838 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.145814 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd9vl\" (UniqueName: \"kubernetes.io/projected/e528f67e-a234-452d-8c81-3e8f2f0d31ac-kube-api-access-qd9vl\") pod \"payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq\" (UID: \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:15.161586 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.161558 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f"] Nov 03 13:17:15.167660 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.167639 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:15.219835 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.219784 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f"] Nov 03 13:17:15.246564 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.246532 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e528f67e-a234-452d-8c81-3e8f2f0d31ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq\" (UID: \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:15.246756 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.246584 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lg5f\" (UniqueName: \"kubernetes.io/projected/79f72dbc-5398-42cd-95b0-1f70910389b2-kube-api-access-5lg5f\") pod \"payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f\" (UID: \"79f72dbc-5398-42cd-95b0-1f70910389b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:15.246756 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.246618 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd9vl\" (UniqueName: \"kubernetes.io/projected/e528f67e-a234-452d-8c81-3e8f2f0d31ac-kube-api-access-qd9vl\") pod \"payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq\" (UID: \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:15.246756 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.246738 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f72dbc-5398-42cd-95b0-1f70910389b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f\" (UID: \"79f72dbc-5398-42cd-95b0-1f70910389b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:15.247335 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.247318 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e528f67e-a234-452d-8c81-3e8f2f0d31ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq\" (UID: \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:15.301500 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.301419 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd9vl\" (UniqueName: \"kubernetes.io/projected/e528f67e-a234-452d-8c81-3e8f2f0d31ac-kube-api-access-qd9vl\") pod \"payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq\" (UID: \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:15.312191 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.312165 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:15.347836 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.347470 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lg5f\" (UniqueName: \"kubernetes.io/projected/79f72dbc-5398-42cd-95b0-1f70910389b2-kube-api-access-5lg5f\") pod \"payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f\" (UID: \"79f72dbc-5398-42cd-95b0-1f70910389b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:15.347836 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.347627 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f72dbc-5398-42cd-95b0-1f70910389b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f\" (UID: \"79f72dbc-5398-42cd-95b0-1f70910389b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:15.348404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.348378 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f72dbc-5398-42cd-95b0-1f70910389b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f\" (UID: \"79f72dbc-5398-42cd-95b0-1f70910389b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:15.402429 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.402396 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lg5f\" (UniqueName: \"kubernetes.io/projected/79f72dbc-5398-42cd-95b0-1f70910389b2-kube-api-access-5lg5f\") pod \"payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f\" (UID: \"79f72dbc-5398-42cd-95b0-1f70910389b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:15.479611 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.479577 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:15.678902 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.678875 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq"] Nov 03 13:17:15.680809 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:15.680762 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode528f67e_a234_452d_8c81_3e8f2f0d31ac.slice/crio-42c26712288fbaffae1f195300938c4645109a044615112e30bf018c6a2ceba5 WatchSource:0}: Error finding container 42c26712288fbaffae1f195300938c4645109a044615112e30bf018c6a2ceba5: Status 404 returned error can't find the container with id 42c26712288fbaffae1f195300938c4645109a044615112e30bf018c6a2ceba5 Nov 03 13:17:15.844124 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:15.844097 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f"] Nov 03 13:17:15.846306 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:15.846280 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f72dbc_5398_42cd_95b0_1f70910389b2.slice/crio-fbe5040ab15f8432aa9de6d4bf50924bb461f5b59342a41cdd0c46f24f7fbfbc WatchSource:0}: Error finding container fbe5040ab15f8432aa9de6d4bf50924bb461f5b59342a41cdd0c46f24f7fbfbc: Status 404 returned error can't find the container with id fbe5040ab15f8432aa9de6d4bf50924bb461f5b59342a41cdd0c46f24f7fbfbc Nov 03 13:17:16.137888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:16.137805 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 13:17:16.194341 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:16.194310 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" event={"ID":"79f72dbc-5398-42cd-95b0-1f70910389b2","Type":"ContainerStarted","Data":"e823d41ea91619745b09a12c4bc60455e578165f96a908feb0b3aa9b58e2d396"} Nov 03 13:17:16.194800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:16.194364 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" event={"ID":"79f72dbc-5398-42cd-95b0-1f70910389b2","Type":"ContainerStarted","Data":"fbe5040ab15f8432aa9de6d4bf50924bb461f5b59342a41cdd0c46f24f7fbfbc"} Nov 03 13:17:16.194800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:16.194395 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:16.195909 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:16.195883 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" event={"ID":"e528f67e-a234-452d-8c81-3e8f2f0d31ac","Type":"ContainerStarted","Data":"171cee89fd955515495805c6edb90cfc943a4dd1fcf346c3094de2b963eafac8"} Nov 03 13:17:16.195975 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:16.195928 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" event={"ID":"e528f67e-a234-452d-8c81-3e8f2f0d31ac","Type":"ContainerStarted","Data":"42c26712288fbaffae1f195300938c4645109a044615112e30bf018c6a2ceba5"} Nov 03 13:17:16.195975 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:16.195949 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:16.353484 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:16.353424 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" podStartSLOduration=2.353411731 podStartE2EDuration="2.353411731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:16.352574341 +0000 UTC m=+767.666546200" watchObservedRunningTime="2026-11-03 13:17:16.353411731 +0000 UTC m=+767.667383634" Nov 03 13:17:16.461356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:16.461287 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" podStartSLOduration=1.461271078 podStartE2EDuration="1.461271078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:16.459455278 +0000 UTC m=+767.773427151" watchObservedRunningTime="2026-11-03 13:17:16.461271078 +0000 UTC m=+767.775242952" Nov 03 13:17:16.888859 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:16.888781 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:17:17.143468 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:17.143385 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 13:17:19.405246 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.405217 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:17:19.488865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.488760 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-data\") pod \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " Nov 03 13:17:19.488865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.488826 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-kube-api-access-44ww2\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-kube-api-access-44ww2\") pod \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " Nov 03 13:17:19.488865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.488851 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-credential-socket\") pod \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " Nov 03 13:17:19.489078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.488887 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-podinfo\") pod \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " Nov 03 13:17:19.489078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.488923 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-envoy\") pod \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " Nov 03 13:17:19.489078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.488964 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-certs\") pod \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " Nov 03 13:17:19.489078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.488988 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istiod-ca-cert\") pod \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " Nov 03 13:17:19.489078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489026 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-socket\") pod \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " Nov 03 13:17:19.489078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489059 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-token\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-token\") pod \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\" (UID: \"7bd25c3f-f82a-4d42-aadb-29f6398f44f1\") " Nov 03 13:17:19.489385 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489102 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-data" pod "7bd25c3f-f82a-4d42-aadb-29f6398f44f1" (UID: "7bd25c3f-f82a-4d42-aadb-29f6398f44f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:19.489385 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489223 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-credential-socket" pod "7bd25c3f-f82a-4d42-aadb-29f6398f44f1" (UID: "7bd25c3f-f82a-4d42-aadb-29f6398f44f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:19.489385 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489338 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-certs" pod "7bd25c3f-f82a-4d42-aadb-29f6398f44f1" (UID: "7bd25c3f-f82a-4d42-aadb-29f6398f44f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:19.489519 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489404 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-socket" pod "7bd25c3f-f82a-4d42-aadb-29f6398f44f1" (UID: "7bd25c3f-f82a-4d42-aadb-29f6398f44f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:19.489519 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489474 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istiod-ca-cert" pod "7bd25c3f-f82a-4d42-aadb-29f6398f44f1" (UID: "7bd25c3f-f82a-4d42-aadb-29f6398f44f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:19.489519 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489511 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:19.489683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489531 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:19.489683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489547 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:19.489683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.489562 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:19.491216 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.491182 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-podinfo" pod "7bd25c3f-f82a-4d42-aadb-29f6398f44f1" (UID: "7bd25c3f-f82a-4d42-aadb-29f6398f44f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:17:19.491355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.491238 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-envoy" pod "7bd25c3f-f82a-4d42-aadb-29f6398f44f1" (UID: "7bd25c3f-f82a-4d42-aadb-29f6398f44f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:19.491355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.491241 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-token" pod "7bd25c3f-f82a-4d42-aadb-29f6398f44f1" (UID: "7bd25c3f-f82a-4d42-aadb-29f6398f44f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:19.491355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.491295 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-kube-api-access-44ww2" pod "7bd25c3f-f82a-4d42-aadb-29f6398f44f1" (UID: "7bd25c3f-f82a-4d42-aadb-29f6398f44f1"). InnerVolumeSpecName "kube-api-access-44ww2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:19.590571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.590467 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44ww2\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-kube-api-access-44ww2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:19.590571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.590507 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:19.590571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.590524 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:19.590571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.590539 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:19.590571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:19.590552 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd25c3f-f82a-4d42-aadb-29f6398f44f1-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:20.194090 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.194060 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 13:17:20.218039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.218007 4541 generic.go:356] "Generic (PLEG): container finished" podID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerID="9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834" exitCode=0 Nov 03 13:17:20.218208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.218058 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" Nov 03 13:17:20.218208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.218093 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" event={"ID":"7bd25c3f-f82a-4d42-aadb-29f6398f44f1","Type":"ContainerDied","Data":"9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834"} Nov 03 13:17:20.218208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.218151 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7" event={"ID":"7bd25c3f-f82a-4d42-aadb-29f6398f44f1","Type":"ContainerDied","Data":"4add471b07cc9a02dfe1a09b5cf57f31cfab688dccc4d064334d10fe2ba4bef9"} Nov 03 13:17:20.218208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.218178 4541 scope.go:122] "RemoveContainer" containerID="9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834" Nov 03 13:17:20.227535 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.227517 4541 scope.go:122] "RemoveContainer" containerID="8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa" Nov 03 13:17:20.235546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.235524 4541 scope.go:122] "RemoveContainer" containerID="9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834" Nov 03 13:17:20.235904 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:20.235879 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834\": container with ID starting with 9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834 not found: ID does not exist" containerID="9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834" Nov 03 13:17:20.236009 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.235909 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834"} err="failed to get container status \"9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834\": rpc error: code = NotFound desc = could not find container \"9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834\": container with ID starting with 9dbe9df3b978cd1bc203c0ca12b4ae200358ce5e739743e7fa79c140cce22834 not found: ID does not exist" Nov 03 13:17:20.236009 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.235928 4541 scope.go:122] "RemoveContainer" containerID="8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa" Nov 03 13:17:20.236227 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:20.236208 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa\": container with ID starting with 8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa not found: ID does not exist" containerID="8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa" Nov 03 13:17:20.236275 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.236235 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa"} err="failed to get container status \"8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa\": rpc error: code = NotFound desc = could not find container \"8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa\": container with ID starting with 8685aecf98b098da4b0b7e1281ce93b0de5e12e01c4e8aeb63988c8281de52fa not found: ID does not exist" Nov 03 13:17:20.413977 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.413930 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck"] Nov 03 13:17:20.419253 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.419225 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7"] Nov 03 13:17:20.461222 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.461140 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a7a9081d-openshift-default-dbf7896f7-jltr7"] Nov 03 13:17:20.869179 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:20.869104 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" path="/var/lib/kubelet/pods/7bd25c3f-f82a-4d42-aadb-29f6398f44f1/volumes" Nov 03 13:17:22.878421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:22.878383 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5"] Nov 03 13:17:22.879001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:22.878981 4541 cpu_manager.go:414] "containerMap: removing container" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" Nov 03 13:17:22.879083 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:22.879003 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" Nov 03 13:17:22.879168 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:22.879155 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" Nov 03 13:17:22.881864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:22.881843 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:22.931364 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:22.931332 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-3c5dbc-serving-cert\"" Nov 03 13:17:22.980305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:22.980262 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5"] Nov 03 13:17:23.024719 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.024687 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.024922 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.024736 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-metrics-certs\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.024922 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.024873 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.025030 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.025005 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qd7m\" (UniqueName: \"kubernetes.io/projected/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-kube-api-access-8qd7m\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.126364 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.126322 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.126559 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.126373 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-metrics-certs\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.126559 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.126401 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.126559 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.126466 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qd7m\" (UniqueName: \"kubernetes.io/projected/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-kube-api-access-8qd7m\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.126748 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:23.126606 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-serving-cert: secret "maas-api-e2e-worker-w4-3c5dbc-serving-cert" not found Nov 03 13:17:23.126748 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:23.126713 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls podName:b09a6b8d-6afc-4e9a-bb05-53d6b7a67383 nodeName:}" failed. No retries permitted until 2026-11-03 13:17:23.626689332 +0000 UTC m=+774.940661189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls") pod "maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" (UID: "b09a6b8d-6afc-4e9a-bb05-53d6b7a67383") : secret "maas-api-e2e-worker-w4-3c5dbc-serving-cert" not found Nov 03 13:17:23.127132 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.127108 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.129049 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.129003 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-metrics-certs\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.182131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.182096 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qd7m\" (UniqueName: \"kubernetes.io/projected/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-kube-api-access-8qd7m\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.631148 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.631111 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.633540 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.633513 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls\") pod \"maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:23.794096 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:23.794053 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:24.160380 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:24.160356 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5"] Nov 03 13:17:24.162449 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:24.162418 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09a6b8d_6afc_4e9a_bb05_53d6b7a67383.slice/crio-1f1cfd3d0425b6be4b39144baabd6d3c4d6566da68dcc66f09b35fe873718b71 WatchSource:0}: Error finding container 1f1cfd3d0425b6be4b39144baabd6d3c4d6566da68dcc66f09b35fe873718b71: Status 404 returned error can't find the container with id 1f1cfd3d0425b6be4b39144baabd6d3c4d6566da68dcc66f09b35fe873718b71 Nov 03 13:17:24.240610 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:24.240569 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" event={"ID":"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383","Type":"ContainerStarted","Data":"1f1cfd3d0425b6be4b39144baabd6d3c4d6566da68dcc66f09b35fe873718b71"} Nov 03 13:17:25.190894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.190788 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 13:17:25.191300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.191015 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 13:17:25.246937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.246898 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" event={"ID":"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383","Type":"ContainerStarted","Data":"df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd"} Nov 03 13:17:25.247096 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.246964 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:25.415110 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.415066 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" podUID="d33a161b-a360-465a-94ab-9e9d74a30318" containerName="payload-processing" containerID="cri-o://cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603" gracePeriod=25 Nov 03 13:17:25.516320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.516252 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" podStartSLOduration=2.5601708199999997 podStartE2EDuration="3.516240245s" totalImagesPullingTime="956.069425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:25.515494839 +0000 UTC m=+776.829466700" watchObservedRunningTime="2026-11-03 13:17:25.516240245 +0000 UTC m=+776.830212105" Nov 03 13:17:25.667310 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.667242 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:17:25.750549 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.750516 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d33a161b-a360-465a-94ab-9e9d74a30318-kube-api-access-9p4p6\" (UniqueName: \"kubernetes.io/projected/d33a161b-a360-465a-94ab-9e9d74a30318-kube-api-access-9p4p6\") pod \"d33a161b-a360-465a-94ab-9e9d74a30318\" (UID: \"d33a161b-a360-465a-94ab-9e9d74a30318\") " Nov 03 13:17:25.750731 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.750636 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d33a161b-a360-465a-94ab-9e9d74a30318-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33a161b-a360-465a-94ab-9e9d74a30318-plugins-config-volume\") pod \"d33a161b-a360-465a-94ab-9e9d74a30318\" (UID: \"d33a161b-a360-465a-94ab-9e9d74a30318\") " Nov 03 13:17:25.751028 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.751005 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33a161b-a360-465a-94ab-9e9d74a30318-plugins-config-volume" pod "d33a161b-a360-465a-94ab-9e9d74a30318" (UID: "d33a161b-a360-465a-94ab-9e9d74a30318"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:25.752645 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.752623 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33a161b-a360-465a-94ab-9e9d74a30318-kube-api-access-9p4p6" pod "d33a161b-a360-465a-94ab-9e9d74a30318" (UID: "d33a161b-a360-465a-94ab-9e9d74a30318"). InnerVolumeSpecName "kube-api-access-9p4p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:25.852114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.852076 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p4p6\" (UniqueName: \"kubernetes.io/projected/d33a161b-a360-465a-94ab-9e9d74a30318-kube-api-access-9p4p6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:25.852114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:25.852112 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d33a161b-a360-465a-94ab-9e9d74a30318-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:26.253326 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.253292 4541 generic.go:356] "Generic (PLEG): container finished" podID="d33a161b-a360-465a-94ab-9e9d74a30318" containerID="cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603" exitCode=0 Nov 03 13:17:26.253847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.253348 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" Nov 03 13:17:26.253847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.253383 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" event={"ID":"d33a161b-a360-465a-94ab-9e9d74a30318","Type":"ContainerDied","Data":"cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603"} Nov 03 13:17:26.253847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.253435 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck" event={"ID":"d33a161b-a360-465a-94ab-9e9d74a30318","Type":"ContainerDied","Data":"450469d6f6cc8ed88068ec08b44a0de917545d77ce2017bcb912adf15e6ff7b8"} Nov 03 13:17:26.253847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.253459 4541 scope.go:122] "RemoveContainer" containerID="cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603" Nov 03 13:17:26.262814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.262795 4541 scope.go:122] "RemoveContainer" containerID="cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603" Nov 03 13:17:26.263101 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:26.263083 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603\": container with ID starting with cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603 not found: ID does not exist" containerID="cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603" Nov 03 13:17:26.263150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.263112 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603"} err="failed to get container status \"cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603\": rpc error: code = NotFound desc = could not find container \"cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603\": container with ID starting with cf990f48e0c51b755a659297085ab0dd6546427c6b3397f666f7c31540138603 not found: ID does not exist" Nov 03 13:17:26.408079 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.408039 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck"] Nov 03 13:17:26.458080 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.458038 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-11ca03ad-7569bccf-lbtck"] Nov 03 13:17:26.867235 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:26.867204 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d33a161b-a360-465a-94ab-9e9d74a30318" path="/var/lib/kubelet/pods/d33a161b-a360-465a-94ab-9e9d74a30318/volumes" Nov 03 13:17:27.204504 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:27.204474 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 13:17:27.204706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:27.204547 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 13:17:28.315449 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.315418 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz"] Nov 03 13:17:28.315878 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.315864 4541 cpu_manager.go:414] "containerMap: removing container" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" Nov 03 13:17:28.315923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.315880 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" Nov 03 13:17:28.315923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.315889 4541 cpu_manager.go:414] "containerMap: removing container" podUID="d33a161b-a360-465a-94ab-9e9d74a30318" containerName="payload-processing" Nov 03 13:17:28.315923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.315895 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d33a161b-a360-465a-94ab-9e9d74a30318" containerName="payload-processing" Nov 03 13:17:28.316032 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.315977 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="d33a161b-a360-465a-94ab-9e9d74a30318" containerName="payload-processing" Nov 03 13:17:28.316032 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.315987 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd25c3f-f82a-4d42-aadb-29f6398f44f1" containerName="istio-proxy" Nov 03 13:17:28.318554 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.318535 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.368726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.368690 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-5bb463-serving-cert\"" Nov 03 13:17:28.417144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.417110 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz"] Nov 03 13:17:28.478942 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.478899 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c103702-fbda-41d3-bf5a-86933fac7f40-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.479140 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.479029 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6sww\" (UniqueName: \"kubernetes.io/projected/5c103702-fbda-41d3-bf5a-86933fac7f40-kube-api-access-c6sww\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.479140 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.479094 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-metrics-certs\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.479251 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.479142 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.580289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.580193 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c103702-fbda-41d3-bf5a-86933fac7f40-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.580289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.580253 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6sww\" (UniqueName: \"kubernetes.io/projected/5c103702-fbda-41d3-bf5a-86933fac7f40-kube-api-access-c6sww\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.580490 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.580300 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-metrics-certs\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.580490 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.580342 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.580939 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.580915 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c103702-fbda-41d3-bf5a-86933fac7f40-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.582792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.582738 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-metrics-certs\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.582871 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.582819 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-maas-api-tls\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.634417 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.634387 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6sww\" (UniqueName: \"kubernetes.io/projected/5c103702-fbda-41d3-bf5a-86933fac7f40-kube-api-access-c6sww\") pod \"maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:28.929964 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:28.929931 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:29.300994 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.300968 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz"] Nov 03 13:17:29.302737 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:29.302705 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c103702_fbda_41d3_bf5a_86933fac7f40.slice/crio-51ddc3ec423b355ae03683caae91709734b20cb37be174c7a3d4b1a16da70067 WatchSource:0}: Error finding container 51ddc3ec423b355ae03683caae91709734b20cb37be174c7a3d4b1a16da70067: Status 404 returned error can't find the container with id 51ddc3ec423b355ae03683caae91709734b20cb37be174c7a3d4b1a16da70067 Nov 03 13:17:29.471430 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.471393 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq"] Nov 03 13:17:29.474703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.474682 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:29.526010 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.525971 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-34b13c\"" Nov 03 13:17:29.526401 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.526370 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-34b13c-dockercfg-7tnmj\"" Nov 03 13:17:29.528322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.528296 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq"] Nov 03 13:17:29.591485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.591409 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg827\" (UniqueName: \"kubernetes.io/projected/7014bbd1-6465-4805-8c57-2bf4b6d98821-kube-api-access-tg827\") pod \"payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq\" (UID: \"7014bbd1-6465-4805-8c57-2bf4b6d98821\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:29.591485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.591465 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7014bbd1-6465-4805-8c57-2bf4b6d98821-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq\" (UID: \"7014bbd1-6465-4805-8c57-2bf4b6d98821\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:29.692853 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.692811 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg827\" (UniqueName: \"kubernetes.io/projected/7014bbd1-6465-4805-8c57-2bf4b6d98821-kube-api-access-tg827\") pod \"payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq\" (UID: \"7014bbd1-6465-4805-8c57-2bf4b6d98821\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:29.693101 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.692876 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7014bbd1-6465-4805-8c57-2bf4b6d98821-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq\" (UID: \"7014bbd1-6465-4805-8c57-2bf4b6d98821\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:29.693515 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.693492 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7014bbd1-6465-4805-8c57-2bf4b6d98821-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq\" (UID: \"7014bbd1-6465-4805-8c57-2bf4b6d98821\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:29.746833 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.746793 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg827\" (UniqueName: \"kubernetes.io/projected/7014bbd1-6465-4805-8c57-2bf4b6d98821-kube-api-access-tg827\") pod \"payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq\" (UID: \"7014bbd1-6465-4805-8c57-2bf4b6d98821\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:29.786680 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:29.786644 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:30.155546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:30.155511 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq"] Nov 03 13:17:30.276567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:30.276525 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" event={"ID":"7014bbd1-6465-4805-8c57-2bf4b6d98821","Type":"ContainerStarted","Data":"58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f"} Nov 03 13:17:30.276794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:30.276594 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" event={"ID":"7014bbd1-6465-4805-8c57-2bf4b6d98821","Type":"ContainerStarted","Data":"cd67250526833798eacc94dad0d4c1efd883eb75c2634043c8123e9d59f8e7df"} Nov 03 13:17:30.276794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:30.276686 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:30.278278 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:30.278252 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" event={"ID":"5c103702-fbda-41d3-bf5a-86933fac7f40","Type":"ContainerStarted","Data":"dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3"} Nov 03 13:17:30.278397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:30.278297 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" event={"ID":"5c103702-fbda-41d3-bf5a-86933fac7f40","Type":"ContainerStarted","Data":"51ddc3ec423b355ae03683caae91709734b20cb37be174c7a3d4b1a16da70067"} Nov 03 13:17:30.278397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:30.278345 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:30.385074 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:30.385010 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" podStartSLOduration=1.384995579 podStartE2EDuration="1.384995579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:30.384169068 +0000 UTC m=+781.698140930" watchObservedRunningTime="2026-11-03 13:17:30.384995579 +0000 UTC m=+781.698967482" Nov 03 13:17:30.496623 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:30.496493 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" podStartSLOduration=1.5937038879999998 podStartE2EDuration="2.496477159s" totalImagesPullingTime="902.773271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:30.496063311 +0000 UTC m=+781.810035171" watchObservedRunningTime="2026-11-03 13:17:30.496477159 +0000 UTC m=+781.810449077" Nov 03 13:17:31.259494 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:31.259463 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 13:17:36.288143 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:36.288108 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 13:17:41.286275 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:41.286240 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 13:17:46.901041 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:17:46.900986 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:17:48.520010 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.519979 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627"] Nov 03 13:17:48.523108 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.523088 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.582277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.582242 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-model-cache\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.582277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.582279 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-home\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.582519 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.582313 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtrck\" (UniqueName: \"kubernetes.io/projected/d6edf898-5a79-4539-a61a-ea6a379c98d5-kube-api-access-vtrck\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.582583 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.582514 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edf898-5a79-4539-a61a-ea6a379c98d5-tls-certs\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.582645 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.582620 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-dshm\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.582698 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.582683 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-tmp-dir\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.619749 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.619714 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"simulator-w2-cdb532-kserve-self-signed-certs\"" Nov 03 13:17:48.619749 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.619721 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"default-dockercfg-zprwm\"" Nov 03 13:17:48.620063 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.619927 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"kube-root-ca.crt\"" Nov 03 13:17:48.620063 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.620052 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"openshift-service-ca.crt\"" Nov 03 13:17:48.622074 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.622054 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627"] Nov 03 13:17:48.684404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.684369 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edf898-5a79-4539-a61a-ea6a379c98d5-tls-certs\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.684601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.684436 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-dshm\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.684601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.684481 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-tmp-dir\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.684715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.684614 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-model-cache\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.684753 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.684717 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-home\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.684818 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.684796 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtrck\" (UniqueName: \"kubernetes.io/projected/d6edf898-5a79-4539-a61a-ea6a379c98d5-kube-api-access-vtrck\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.685244 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.685216 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-tmp-dir\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.685358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.685337 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-home\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.685500 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.685482 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-model-cache\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.686990 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.686970 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edf898-5a79-4539-a61a-ea6a379c98d5-tls-certs\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.687473 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.687456 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-dshm\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.741355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.741319 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtrck\" (UniqueName: \"kubernetes.io/projected/d6edf898-5a79-4539-a61a-ea6a379c98d5-kube-api-access-vtrck\") pod \"simulator-w2-cdb532-kserve-5c5d7df75b-gn627\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:48.834016 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:48.833920 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:49.204016 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:49.203985 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627"] Nov 03 13:17:49.205404 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:17:49.205372 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6edf898_5a79_4539_a61a_ea6a379c98d5.slice/crio-a8f89cca2e60b08cbec29f360656ac5c9cca424e46d7c735d2cb2e49a241c8a7 WatchSource:0}: Error finding container a8f89cca2e60b08cbec29f360656ac5c9cca424e46d7c735d2cb2e49a241c8a7: Status 404 returned error can't find the container with id a8f89cca2e60b08cbec29f360656ac5c9cca424e46d7c735d2cb2e49a241c8a7 Nov 03 13:17:49.370743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:49.370702 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" event={"ID":"d6edf898-5a79-4539-a61a-ea6a379c98d5","Type":"ContainerStarted","Data":"b23e9d7ad87bb86d1d97cea410a78d17316bac6509f7d4b95a59ecf329653c7f"} Nov 03 13:17:49.370743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:49.370745 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" event={"ID":"d6edf898-5a79-4539-a61a-ea6a379c98d5","Type":"ContainerStarted","Data":"a8f89cca2e60b08cbec29f360656ac5c9cca424e46d7c735d2cb2e49a241c8a7"} Nov 03 13:17:49.480289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:49.480169 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" podStartSLOduration=1.480153831 podStartE2EDuration="1.480153831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:49.480070649 +0000 UTC m=+800.794042510" watchObservedRunningTime="2026-11-03 13:17:49.480153831 +0000 UTC m=+800.794125690" Nov 03 13:17:49.835024 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:49.834938 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:58.834858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:58.834823 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:58.847590 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:58.847561 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:17:59.431336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:17:59.431306 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 13:18:08.257269 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.257231 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq"] Nov 03 13:18:08.263756 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.263731 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.314284 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.314249 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq"] Nov 03 13:18:08.358783 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.358742 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-42c707-serving-cert\"" Nov 03 13:18:08.383669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.383635 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-metrics-certs\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.383858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.383681 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.383858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.383703 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd08615b-547e-4e1d-a572-70f216d10e41-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.383858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.383843 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74ll\" (UniqueName: \"kubernetes.io/projected/cd08615b-547e-4e1d-a572-70f216d10e41-kube-api-access-r74ll\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.485230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.485196 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-metrics-certs\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.485437 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.485242 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.485437 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:18:08.485328 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-serving-cert: secret "maas-api-e2e-worker-w0-42c707-serving-cert" not found Nov 03 13:18:08.485437 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.485362 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd08615b-547e-4e1d-a572-70f216d10e41-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.485437 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:18:08.485390 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls podName:cd08615b-547e-4e1d-a572-70f216d10e41 nodeName:}" failed. No retries permitted until 2026-11-03 13:18:08.985376528 +0000 UTC m=+820.299348367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls") pod "maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" (UID: "cd08615b-547e-4e1d-a572-70f216d10e41") : secret "maas-api-e2e-worker-w0-42c707-serving-cert" not found Nov 03 13:18:08.485437 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.485424 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74ll\" (UniqueName: \"kubernetes.io/projected/cd08615b-547e-4e1d-a572-70f216d10e41-kube-api-access-r74ll\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.486062 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.486039 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd08615b-547e-4e1d-a572-70f216d10e41-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.487986 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.487966 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-metrics-certs\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.542667 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.542587 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74ll\" (UniqueName: \"kubernetes.io/projected/cd08615b-547e-4e1d-a572-70f216d10e41-kube-api-access-r74ll\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.991450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.991406 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:08.994018 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:08.993992 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls\") pod \"maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:09.180649 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:09.180609 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:09.569703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:09.569675 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq"] Nov 03 13:18:09.571548 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:18:09.571517 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd08615b_547e_4e1d_a572_70f216d10e41.slice/crio-33fa8cd980bf97570cadc0d1b25f35707dc28acddf2b4e5eebf597d09be1fdab WatchSource:0}: Error finding container 33fa8cd980bf97570cadc0d1b25f35707dc28acddf2b4e5eebf597d09be1fdab: Status 404 returned error can't find the container with id 33fa8cd980bf97570cadc0d1b25f35707dc28acddf2b4e5eebf597d09be1fdab Nov 03 13:18:10.484176 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:10.484145 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" event={"ID":"cd08615b-547e-4e1d-a572-70f216d10e41","Type":"ContainerStarted","Data":"33fa8cd980bf97570cadc0d1b25f35707dc28acddf2b4e5eebf597d09be1fdab"} Nov 03 13:18:11.490478 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:11.490437 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" event={"ID":"cd08615b-547e-4e1d-a572-70f216d10e41","Type":"ContainerStarted","Data":"8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f"} Nov 03 13:18:11.490928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:11.490622 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:11.598992 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:11.598933 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" podStartSLOduration=2.697407746 podStartE2EDuration="3.598913918s" totalImagesPullingTime="901.506172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:11.598235664 +0000 UTC m=+822.912207524" watchObservedRunningTime="2026-11-03 13:18:11.598913918 +0000 UTC m=+822.912885781" Nov 03 13:18:12.986341 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:12.986307 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn"] Nov 03 13:18:12.990155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:12.990128 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.024811 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.024754 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-tmp-dir\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.024976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.024845 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3660cd0d-de10-4932-8d1e-764392f3e005-tls-certs\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.024976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.024876 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-home\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.024976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.024969 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-dshm\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.025112 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.025009 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-model-cache\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.025147 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.025114 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwtk\" (UniqueName: \"kubernetes.io/projected/3660cd0d-de10-4932-8d1e-764392f3e005-kube-api-access-9lwtk\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.090697 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.090661 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"default-dockercfg-4g6gg\"" Nov 03 13:18:13.090697 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.090667 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"openshift-service-ca.crt\"" Nov 03 13:18:13.090904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.090665 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"simulator-w4-3c5dbc-kserve-self-signed-certs\"" Nov 03 13:18:13.091111 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.091094 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"kube-root-ca.crt\"" Nov 03 13:18:13.093951 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.093924 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn"] Nov 03 13:18:13.126331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.126300 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwtk\" (UniqueName: \"kubernetes.io/projected/3660cd0d-de10-4932-8d1e-764392f3e005-kube-api-access-9lwtk\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.126543 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.126353 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-tmp-dir\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.126543 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.126453 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3660cd0d-de10-4932-8d1e-764392f3e005-tls-certs\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.126543 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.126479 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-home\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.126543 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.126509 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-dshm\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.126804 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.126636 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-model-cache\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.126873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.126851 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-tmp-dir\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.126933 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.126910 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-home\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.127044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.127021 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-model-cache\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.128743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.128717 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-dshm\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.129024 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.129008 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3660cd0d-de10-4932-8d1e-764392f3e005-tls-certs\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.180324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.180292 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwtk\" (UniqueName: \"kubernetes.io/projected/3660cd0d-de10-4932-8d1e-764392f3e005-kube-api-access-9lwtk\") pod \"simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.302659 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.302569 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:13.668460 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:13.668431 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn"] Nov 03 13:18:13.669967 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:18:13.669937 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3660cd0d_de10_4932_8d1e_764392f3e005.slice/crio-f94ca35cc43fd5db651590bd8502f8e4694f004b49bc4be7bbc0cc490d1d8cbd WatchSource:0}: Error finding container f94ca35cc43fd5db651590bd8502f8e4694f004b49bc4be7bbc0cc490d1d8cbd: Status 404 returned error can't find the container with id f94ca35cc43fd5db651590bd8502f8e4694f004b49bc4be7bbc0cc490d1d8cbd Nov 03 13:18:14.505686 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:14.505653 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" event={"ID":"3660cd0d-de10-4932-8d1e-764392f3e005","Type":"ContainerStarted","Data":"f622d6cf08611f54667880f27bf1322b1b635fce09bf6d151e28a639095b9802"} Nov 03 13:18:14.506114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:14.505697 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" event={"ID":"3660cd0d-de10-4932-8d1e-764392f3e005","Type":"ContainerStarted","Data":"f94ca35cc43fd5db651590bd8502f8e4694f004b49bc4be7bbc0cc490d1d8cbd"} Nov 03 13:18:14.614424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:14.614364 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" podStartSLOduration=2.614349181 podStartE2EDuration="2.614349181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:14.61345689 +0000 UTC m=+825.927428751" watchObservedRunningTime="2026-11-03 13:18:14.614349181 +0000 UTC m=+825.928321040" Nov 03 13:18:15.303436 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:15.303403 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:16.889298 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:18:16.889257 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:18:17.501172 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:17.501148 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 13:18:23.303289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:23.303249 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:23.315902 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:23.315876 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:23.558706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:23.558621 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:18:26.643227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.643196 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq"] Nov 03 13:18:26.647382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.647356 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.746552 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.746514 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-dshm\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.746704 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.746559 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824d587c-60d2-4cf2-9c31-37f8baf60308-tls-certs\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.746704 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.746593 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-model-cache\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.746704 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.746634 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cflbk\" (UniqueName: \"kubernetes.io/projected/824d587c-60d2-4cf2-9c31-37f8baf60308-kube-api-access-cflbk\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.746846 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.746725 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-home\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.746892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.746853 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-tmp-dir\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.749068 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.749036 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"simulator-w3-5bb463-kserve-self-signed-certs\"" Nov 03 13:18:26.749384 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.749369 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"openshift-service-ca.crt\"" Nov 03 13:18:26.749779 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.749753 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"default-dockercfg-bhrh5\"" Nov 03 13:18:26.751319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.751301 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq"] Nov 03 13:18:26.753145 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.753126 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"kube-root-ca.crt\"" Nov 03 13:18:26.848154 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.848115 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-tmp-dir\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.848342 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.848165 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-dshm\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.848342 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.848193 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824d587c-60d2-4cf2-9c31-37f8baf60308-tls-certs\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.848342 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.848215 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-model-cache\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.848342 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.848231 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cflbk\" (UniqueName: \"kubernetes.io/projected/824d587c-60d2-4cf2-9c31-37f8baf60308-kube-api-access-cflbk\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.848342 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.848258 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-home\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.848661 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.848642 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-tmp-dir\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.848743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.848708 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-model-cache\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.848743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.848726 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-home\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.850522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.850502 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-dshm\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.850718 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.850702 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824d587c-60d2-4cf2-9c31-37f8baf60308-tls-certs\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.912379 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.912346 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cflbk\" (UniqueName: \"kubernetes.io/projected/824d587c-60d2-4cf2-9c31-37f8baf60308-kube-api-access-cflbk\") pod \"simulator-w3-5bb463-kserve-b8bff58fb-dwvqq\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:26.962286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:26.962249 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:27.327624 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:27.327594 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq"] Nov 03 13:18:27.329225 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:18:27.329200 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod824d587c_60d2_4cf2_9c31_37f8baf60308.slice/crio-b2c208addb2bb4e95e5376bf125aaee11d15a8b423dd70e7359f3c72bb919bfb WatchSource:0}: Error finding container b2c208addb2bb4e95e5376bf125aaee11d15a8b423dd70e7359f3c72bb919bfb: Status 404 returned error can't find the container with id b2c208addb2bb4e95e5376bf125aaee11d15a8b423dd70e7359f3c72bb919bfb Nov 03 13:18:27.571115 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:27.569240 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" event={"ID":"824d587c-60d2-4cf2-9c31-37f8baf60308","Type":"ContainerStarted","Data":"b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085"} Nov 03 13:18:27.571115 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:27.569289 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" event={"ID":"824d587c-60d2-4cf2-9c31-37f8baf60308","Type":"ContainerStarted","Data":"b2c208addb2bb4e95e5376bf125aaee11d15a8b423dd70e7359f3c72bb919bfb"} Nov 03 13:18:27.680492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:27.680436 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" podStartSLOduration=1.68042139 podStartE2EDuration="1.68042139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:27.678827346 +0000 UTC m=+838.992799206" watchObservedRunningTime="2026-11-03 13:18:27.68042139 +0000 UTC m=+838.994393250" Nov 03 13:18:27.963201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:27.963122 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:36.962634 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:36.962594 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:36.975204 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:36.975178 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:37.629264 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:37.629234 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:18:40.135219 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.135130 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv"] Nov 03 13:18:40.140175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.140149 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.244360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.244328 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-cdb532\"/\"premium-w2-cdb532-kserve-self-signed-certs\"" Nov 03 13:18:40.246706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.246682 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv"] Nov 03 13:18:40.285659 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.285623 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-home\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.285838 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.285684 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tmp-dir\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.285888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.285823 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-model-cache\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.285888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.285868 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-dshm\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.285968 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.285947 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tls-certs\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.286027 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.286010 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5js2b\" (UniqueName: \"kubernetes.io/projected/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-kube-api-access-5js2b\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.386716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.386627 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tmp-dir\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.386716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.386676 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-model-cache\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.386716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.386697 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-dshm\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.386986 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.386717 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tls-certs\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.386986 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.386760 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5js2b\" (UniqueName: \"kubernetes.io/projected/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-kube-api-access-5js2b\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.386986 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.386878 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-home\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.387192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.387166 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tmp-dir\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.387259 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.387215 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-model-cache\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.387259 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.387248 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-home\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.389179 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.389158 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-dshm\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.389363 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.389348 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tls-certs\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.441320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.441283 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5js2b\" (UniqueName: \"kubernetes.io/projected/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-kube-api-access-5js2b\") pod \"premium-w2-cdb532-kserve-688f9985bf-jf8qv\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.482335 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.482297 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:40.853644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:40.853615 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv"] Nov 03 13:18:40.856062 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:18:40.856035 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd74e0ba8_a137_4e34_9a7d_a7c4559fce1b.slice/crio-a8ba9143a544f220990995cc4af0081c63f6d2f9abe740012efe66c113f84a90 WatchSource:0}: Error finding container a8ba9143a544f220990995cc4af0081c63f6d2f9abe740012efe66c113f84a90: Status 404 returned error can't find the container with id a8ba9143a544f220990995cc4af0081c63f6d2f9abe740012efe66c113f84a90 Nov 03 13:18:41.642605 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:41.642565 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" event={"ID":"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b","Type":"ContainerStarted","Data":"47d200e2ee3aa4db69f52b70d45baf4d9ddefe803897dd0ae60a9c62f60ed08e"} Nov 03 13:18:41.642605 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:41.642609 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" event={"ID":"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b","Type":"ContainerStarted","Data":"a8ba9143a544f220990995cc4af0081c63f6d2f9abe740012efe66c113f84a90"} Nov 03 13:18:41.755149 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:41.755092 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" podStartSLOduration=1.7550789839999998 podStartE2EDuration="1.755078984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:18:41.753716839 +0000 UTC m=+853.067688699" watchObservedRunningTime="2026-11-03 13:18:41.755078984 +0000 UTC m=+853.069050837" Nov 03 13:18:42.483105 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:42.483065 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:46.890413 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:18:46.890366 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:18:50.482767 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:50.482721 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:50.495811 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:50.495763 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:18:50.699017 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:18:50.698986 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:19:16.891057 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:19:16.891016 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:19:22.048680 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.048595 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb"] Nov 03 13:19:22.052385 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.052361 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.114478 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.114439 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafbdc79-6bde-48a0-853c-8a92f80481eb-tls-certs\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.114631 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.114549 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-tmp-dir\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.114631 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.114601 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-model-cache\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.114728 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.114637 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-dshm\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.114728 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.114680 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8tlj\" (UniqueName: \"kubernetes.io/projected/aafbdc79-6bde-48a0-853c-8a92f80481eb-kube-api-access-n8tlj\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.114866 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.114753 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-home\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.148877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.148845 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"default-dockercfg-mzpk2\"" Nov 03 13:19:22.148877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.148845 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"simulator-w0-42c707-kserve-self-signed-certs\"" Nov 03 13:19:22.149132 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.148974 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"kube-root-ca.crt\"" Nov 03 13:19:22.149210 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.149194 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"openshift-service-ca.crt\"" Nov 03 13:19:22.151194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.151176 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb"] Nov 03 13:19:22.215749 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.215714 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-tmp-dir\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.215975 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.215761 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-model-cache\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.215975 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.215799 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-dshm\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.215975 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.215824 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8tlj\" (UniqueName: \"kubernetes.io/projected/aafbdc79-6bde-48a0-853c-8a92f80481eb-kube-api-access-n8tlj\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.215975 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.215881 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-home\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.215975 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.215909 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafbdc79-6bde-48a0-853c-8a92f80481eb-tls-certs\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.216322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.216298 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-model-cache\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.216379 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.216327 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-tmp-dir\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.216379 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.216343 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-home\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.218127 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.218102 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-dshm\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.218294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.218277 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafbdc79-6bde-48a0-853c-8a92f80481eb-tls-certs\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.271762 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.271725 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8tlj\" (UniqueName: \"kubernetes.io/projected/aafbdc79-6bde-48a0-853c-8a92f80481eb-kube-api-access-n8tlj\") pod \"simulator-w0-42c707-kserve-7c4dbd844-s5qqb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.365255 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.365172 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:22.730969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.730942 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb"] Nov 03 13:19:22.733088 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:19:22.733058 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaafbdc79_6bde_48a0_853c_8a92f80481eb.slice/crio-3c72d089f58719d3fbe185d58ca9f6d9ef5e8678326a21594ab5e6abfd2fa6f3 WatchSource:0}: Error finding container 3c72d089f58719d3fbe185d58ca9f6d9ef5e8678326a21594ab5e6abfd2fa6f3: Status 404 returned error can't find the container with id 3c72d089f58719d3fbe185d58ca9f6d9ef5e8678326a21594ab5e6abfd2fa6f3 Nov 03 13:19:22.846488 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.846452 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" event={"ID":"aafbdc79-6bde-48a0-853c-8a92f80481eb","Type":"ContainerStarted","Data":"b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb"} Nov 03 13:19:22.846693 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.846498 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" event={"ID":"aafbdc79-6bde-48a0-853c-8a92f80481eb","Type":"ContainerStarted","Data":"3c72d089f58719d3fbe185d58ca9f6d9ef5e8678326a21594ab5e6abfd2fa6f3"} Nov 03 13:19:22.960712 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:22.960657 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" podStartSLOduration=0.960643145 podStartE2EDuration="960.643145ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:22.958603228 +0000 UTC m=+894.272575102" watchObservedRunningTime="2026-11-03 13:19:22.960643145 +0000 UTC m=+894.274615005" Nov 03 13:19:23.366072 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:23.366033 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:28.972362 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:28.972328 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:19:28.973284 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:28.973264 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:19:32.365617 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:32.365581 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:32.378318 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:32.378294 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:32.908052 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:32.908024 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:19:46.889627 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:19:46.889589 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:19:51.555707 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.555664 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp"] Nov 03 13:19:51.559743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.559720 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.648155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.648118 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tls-certs\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.648345 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.648175 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-dshm\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.648399 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.648362 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-model-cache\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.648500 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.648479 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-home\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.648587 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.648571 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tmp-dir\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.648638 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.648622 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbpgb\" (UniqueName: \"kubernetes.io/projected/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-kube-api-access-pbpgb\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.658433 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.658401 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"openshift-service-ca.crt\"" Nov 03 13:19:51.658433 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.658405 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"kube-root-ca.crt\"" Nov 03 13:19:51.658634 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.658468 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"default-dockercfg-jc47d\"" Nov 03 13:19:51.658692 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.658676 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"premium-w1-34b13c-kserve-self-signed-certs\"" Nov 03 13:19:51.660743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.660721 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp"] Nov 03 13:19:51.750008 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.749964 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-model-cache\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.750207 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.750054 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-home\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.750207 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.750123 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tmp-dir\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.750207 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.750167 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbpgb\" (UniqueName: \"kubernetes.io/projected/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-kube-api-access-pbpgb\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.750373 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.750208 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tls-certs\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.750373 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.750243 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-dshm\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.750477 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.750414 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-model-cache\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.750477 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.750435 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-home\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.750587 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.750485 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tmp-dir\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.752554 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.752532 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-dshm\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.752859 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.752841 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tls-certs\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.808162 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.808084 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbpgb\" (UniqueName: \"kubernetes.io/projected/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-kube-api-access-pbpgb\") pod \"premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:51.871827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:51.871789 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:19:52.244760 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:52.244716 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp"] Nov 03 13:19:52.247209 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:19:52.247182 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33d2c9a3_a337_406d_a9ed_0d5d7e4bcfe0.slice/crio-35adb7681b9ae7b8938c9cdd69b39a3340993a6b0a6201a7efa81bb0482aa35b WatchSource:0}: Error finding container 35adb7681b9ae7b8938c9cdd69b39a3340993a6b0a6201a7efa81bb0482aa35b: Status 404 returned error can't find the container with id 35adb7681b9ae7b8938c9cdd69b39a3340993a6b0a6201a7efa81bb0482aa35b Nov 03 13:19:53.010879 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:53.010846 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" event={"ID":"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0","Type":"ContainerStarted","Data":"523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818"} Nov 03 13:19:53.011258 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:53.010887 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" event={"ID":"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0","Type":"ContainerStarted","Data":"35adb7681b9ae7b8938c9cdd69b39a3340993a6b0a6201a7efa81bb0482aa35b"} Nov 03 13:19:53.121270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:53.121216 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" podStartSLOduration=2.121201961 podStartE2EDuration="2.121201961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:53.119221328 +0000 UTC m=+924.433193195" watchObservedRunningTime="2026-11-03 13:19:53.121201961 +0000 UTC m=+924.435173836" Nov 03 13:19:53.872143 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:19:53.872095 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:20:01.872385 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:01.872351 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:20:01.885573 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:01.885544 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:20:02.066255 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:02.066224 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:20:16.890279 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:20:16.890204 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:20:39.593354 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.593320 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8"] Nov 03 13:20:39.597639 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.597619 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.647051 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.647021 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cdb532-openshift-default-dockercfg-q2fdz\"" Nov 03 13:20:39.696814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.696754 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8"] Nov 03 13:20:39.754651 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.754615 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.754651 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.754654 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.754925 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.754674 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b07c5e28-99e1-470a-a007-889ba3917657-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.754925 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.754738 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.754925 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.754889 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.755093 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.754965 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.755093 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.755019 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbfsl\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-kube-api-access-bbfsl\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.755093 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.755068 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.755208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.755134 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b07c5e28-99e1-470a-a007-889ba3917657-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.855820 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.855703 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbfsl\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-kube-api-access-bbfsl\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.855820 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.855757 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.855820 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.855813 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b07c5e28-99e1-470a-a007-889ba3917657-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856113 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.855877 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856113 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.855897 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856113 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.855912 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b07c5e28-99e1-470a-a007-889ba3917657-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856113 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.855939 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856113 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.856010 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856113 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.856064 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856409 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.856316 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856409 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.856387 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856506 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.856472 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856547 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.856521 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.856691 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.856671 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b07c5e28-99e1-470a-a007-889ba3917657-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.858725 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.858698 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.858964 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.858947 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b07c5e28-99e1-470a-a007-889ba3917657-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.910112 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.910075 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.910112 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.910110 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbfsl\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-kube-api-access-bbfsl\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:39.910881 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:39.910862 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:40.328959 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:40.328931 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8"] Nov 03 13:20:40.330474 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:20:40.330433 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07c5e28_99e1_470a_a007_889ba3917657.slice/crio-3cb3a94ba98111e7ff2041729254892010f75eea2b768c5f76b60e3d98437bdd WatchSource:0}: Error finding container 3cb3a94ba98111e7ff2041729254892010f75eea2b768c5f76b60e3d98437bdd: Status 404 returned error can't find the container with id 3cb3a94ba98111e7ff2041729254892010f75eea2b768c5f76b60e3d98437bdd Nov 03 13:20:40.332799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:40.332748 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:20:40.332888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:40.332828 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:20:40.332927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:40.332889 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:20:41.268556 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:41.268519 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" event={"ID":"b07c5e28-99e1-470a-a007-889ba3917657","Type":"ContainerStarted","Data":"e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8"} Nov 03 13:20:41.268976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:41.268605 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" event={"ID":"b07c5e28-99e1-470a-a007-889ba3917657","Type":"ContainerStarted","Data":"3cb3a94ba98111e7ff2041729254892010f75eea2b768c5f76b60e3d98437bdd"} Nov 03 13:20:41.428095 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:41.428040 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" podStartSLOduration=2.428026292 podStartE2EDuration="2.428026292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:41.425236275 +0000 UTC m=+972.739208135" watchObservedRunningTime="2026-11-03 13:20:41.428026292 +0000 UTC m=+972.741998151" Nov 03 13:20:41.911606 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:41.911576 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:41.913256 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:41.913229 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:41.913384 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:41.913280 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 13:20:42.911463 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:42.911429 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:42.911884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:42.911481 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 13:20:43.911758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:43.911721 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:43.912271 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:43.911840 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 13:20:44.911435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:44.911402 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:44.911620 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:44.911452 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 13:20:45.911811 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:45.911754 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 03 13:20:45.912216 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:45.911843 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 03 13:20:46.893021 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:20:46.892985 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:20:47.824105 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:47.824074 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:47.825101 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:47.825072 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:20:47.825951 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:20:47.825933 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 13:21:16.889289 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:21:16.889252 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:21:21.417956 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:21.417911 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:21.418376 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:21.417991 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:43.189014 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.188926 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk"] Nov 03 13:21:43.193641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.193617 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.332936 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.332902 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.332936 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.332949 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.333158 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.332984 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.333158 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.333049 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.333240 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.333201 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b671863-3595-4ae5-a3fa-b8362ec220fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.333285 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.333247 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.333323 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.333306 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8vfq\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-kube-api-access-w8vfq\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.333366 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.333351 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.333416 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.333401 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.393758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.393717 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk"] Nov 03 13:21:43.434539 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.434502 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8vfq\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-kube-api-access-w8vfq\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.434741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.434562 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.434741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.434620 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.434741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.434674 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.434741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.434713 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.434992 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.434767 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.434992 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.434820 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.434992 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.434902 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b671863-3595-4ae5-a3fa-b8362ec220fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.435142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.435045 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.435142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.435083 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.435142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.435103 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.435329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.435306 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.435392 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.435358 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.435647 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.435624 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b671863-3595-4ae5-a3fa-b8362ec220fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.436962 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.436937 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.437270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.437251 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.489628 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.489552 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.489797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.489755 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8vfq\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-kube-api-access-w8vfq\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8hcfk\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.506627 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.506597 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:43.922935 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.922911 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk"] Nov 03 13:21:43.924496 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:21:43.924468 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b671863_3595_4ae5_a3fa_b8362ec220fa.slice/crio-c39202922967f6d5f4a8e593b471a0477016326c0d8edec8cfb5b8e1efeea820 WatchSource:0}: Error finding container c39202922967f6d5f4a8e593b471a0477016326c0d8edec8cfb5b8e1efeea820: Status 404 returned error can't find the container with id c39202922967f6d5f4a8e593b471a0477016326c0d8edec8cfb5b8e1efeea820 Nov 03 13:21:43.926880 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.926847 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:43.926946 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.926915 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:43.926994 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:43.926944 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:21:44.653794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:44.642158 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" event={"ID":"0b671863-3595-4ae5-a3fa-b8362ec220fa","Type":"ContainerStarted","Data":"8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92"} Nov 03 13:21:44.653794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:44.642231 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" event={"ID":"0b671863-3595-4ae5-a3fa-b8362ec220fa","Type":"ContainerStarted","Data":"c39202922967f6d5f4a8e593b471a0477016326c0d8edec8cfb5b8e1efeea820"} Nov 03 13:21:44.812011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:44.811956 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podStartSLOduration=1.811941221 podStartE2EDuration="1.811941221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:44.809407666 +0000 UTC m=+1036.123379527" watchObservedRunningTime="2026-11-03 13:21:44.811941221 +0000 UTC m=+1036.125913081" Nov 03 13:21:45.507599 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:45.507565 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:45.509262 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:45.509235 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:21:45.509385 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:45.509282 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 13:21:46.507928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:46.507890 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:21:46.508329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:46.507958 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 13:21:46.889447 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:21:46.889363 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:21:47.507001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:47.506972 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:21:47.507193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:47.507020 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 13:21:48.507683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:48.507642 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:21:48.508306 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:48.507699 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 13:21:49.507523 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:49.507483 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:21:49.507711 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:49.507552 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 13:21:50.507197 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:50.507157 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:21:50.507370 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:50.507233 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 13:21:51.507945 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:51.507903 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 03 13:21:51.508340 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:51.507989 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 03 13:21:52.511855 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:52.511824 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:52.512237 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:52.512199 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:21:52.512935 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:21:52.512916 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:22:00.667208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.667163 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt"] Nov 03 13:22:00.671613 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.671583 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.731125 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.731094 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-home\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.731294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.731134 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tmp-dir\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.731294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.731161 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-model-cache\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.731392 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.731361 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmrw\" (UniqueName: \"kubernetes.io/projected/582feb39-8aa5-43b6-a68c-6850b5cec3e5-kube-api-access-jdmrw\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.731448 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.731431 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-dshm\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.731483 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.731469 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tls-certs\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.769899 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.769797 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"distinct-w4-3c5dbc-kserve-self-signed-certs\"" Nov 03 13:22:00.771985 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.771961 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt"] Nov 03 13:22:00.832377 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.832344 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-dshm\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.832377 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.832381 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tls-certs\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.832642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.832467 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-home\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.832642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.832497 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tmp-dir\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.832642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.832530 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-model-cache\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.832642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.832587 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmrw\" (UniqueName: \"kubernetes.io/projected/582feb39-8aa5-43b6-a68c-6850b5cec3e5-kube-api-access-jdmrw\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.833025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.832987 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tmp-dir\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.833328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.833303 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-home\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.833492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.833321 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-model-cache\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.835973 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.835944 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-dshm\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.836153 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.836135 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tls-certs\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.887747 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.887715 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmrw\" (UniqueName: \"kubernetes.io/projected/582feb39-8aa5-43b6-a68c-6850b5cec3e5-kube-api-access-jdmrw\") pod \"distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:00.984450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:00.984355 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:01.365293 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:01.365261 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt"] Nov 03 13:22:01.365865 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:22:01.365836 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod582feb39_8aa5_43b6_a68c_6850b5cec3e5.slice/crio-7aa99186b75144580cbaecbe15c8fe5a707cd382b5d0ddbeb3f7853231448b9b WatchSource:0}: Error finding container 7aa99186b75144580cbaecbe15c8fe5a707cd382b5d0ddbeb3f7853231448b9b: Status 404 returned error can't find the container with id 7aa99186b75144580cbaecbe15c8fe5a707cd382b5d0ddbeb3f7853231448b9b Nov 03 13:22:01.745799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:01.745730 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" event={"ID":"582feb39-8aa5-43b6-a68c-6850b5cec3e5","Type":"ContainerStarted","Data":"98f3ede2256abc5547f5105c1889a8773e788709de46768e406f5a57e9e7dda1"} Nov 03 13:22:01.745799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:01.745799 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" event={"ID":"582feb39-8aa5-43b6-a68c-6850b5cec3e5","Type":"ContainerStarted","Data":"7aa99186b75144580cbaecbe15c8fe5a707cd382b5d0ddbeb3f7853231448b9b"} Nov 03 13:22:01.856406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:01.856334 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" podStartSLOduration=1.856315079 podStartE2EDuration="1.856315079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:01.854156868 +0000 UTC m=+1053.168128722" watchObservedRunningTime="2026-11-03 13:22:01.856315079 +0000 UTC m=+1053.170286944" Nov 03 13:22:01.985021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:01.984986 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:10.984579 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:10.984532 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:10.997925 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:10.997892 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:11.810857 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:11.810823 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 13:22:16.890061 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:22:16.890023 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:22:18.042797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.042750 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd"] Nov 03 13:22:18.046450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.046432 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.097183 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.097152 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-34b13c-openshift-default-dockercfg-kf4m8\"" Nov 03 13:22:18.146284 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.146252 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd"] Nov 03 13:22:18.211890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.211856 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvndg\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-kube-api-access-qvndg\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.212057 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.211898 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.212057 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.211968 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.212057 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.212026 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.212166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.212102 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.212166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.212147 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.212228 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.212175 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.212273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.212226 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.212273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.212250 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.313792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.313672 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvndg\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-kube-api-access-qvndg\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.313792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.313727 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.313792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.313753 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.313827 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.313873 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.313918 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.313957 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.314002 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.314041 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314426 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.314290 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314426 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.314354 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.314507 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314565 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.314551 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.314626 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.314609 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.316607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.316585 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.316975 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.316951 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.368274 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.368228 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.368565 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.368544 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvndg\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-kube-api-access-qvndg\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:18.659807 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:18.659754 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:19.077449 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:19.077424 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd"] Nov 03 13:22:19.079116 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:22:19.079084 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f98690_9c1c_462c_b1d2_a9d4a5ef11b8.slice/crio-074538f62cca09f2bab3a3a2db67fb54aed994ee60d4e52af125a67ee9fdd338 WatchSource:0}: Error finding container 074538f62cca09f2bab3a3a2db67fb54aed994ee60d4e52af125a67ee9fdd338: Status 404 returned error can't find the container with id 074538f62cca09f2bab3a3a2db67fb54aed994ee60d4e52af125a67ee9fdd338 Nov 03 13:22:19.081529 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:19.081497 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:22:19.081608 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:19.081560 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:22:19.081608 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:19.081591 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:22:19.856122 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:19.856080 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" event={"ID":"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8","Type":"ContainerStarted","Data":"04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98"} Nov 03 13:22:19.856554 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:19.856524 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" event={"ID":"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8","Type":"ContainerStarted","Data":"074538f62cca09f2bab3a3a2db67fb54aed994ee60d4e52af125a67ee9fdd338"} Nov 03 13:22:20.660844 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:20.660798 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:20.662989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:20.662956 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:20.663126 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:20.663008 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 13:22:21.660367 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:21.660327 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:21.660596 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:21.660392 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 13:22:22.660728 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:22.660685 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:22.661165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:22.660760 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 13:22:23.660527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:23.660488 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:23.660730 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:23.660571 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 13:22:24.660331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:24.660292 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:24.660865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:24.660362 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 13:22:25.660404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:25.660367 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Nov 03 13:22:25.660841 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:25.660429 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Nov 03 13:22:26.664715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:26.664685 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:26.665185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:26.665163 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:26.666004 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:26.665984 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:22:26.862414 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:22:26.862350 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" podStartSLOduration=8.862333472 podStartE2EDuration="8.862333472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:20.070061499 +0000 UTC m=+1071.384033358" watchObservedRunningTime="2026-11-03 13:22:26.862333472 +0000 UTC m=+1078.176305335" Nov 03 13:22:46.890107 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:22:46.890070 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:23:16.890305 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:23:16.890223 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:23:17.675522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.675488 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg"] Nov 03 13:23:17.679243 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.679223 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.734367 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.734329 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlw5s\" (UniqueName: \"kubernetes.io/projected/cca463a2-02b8-46a2-8a66-fd398b50eafd-kube-api-access-nlw5s\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.734541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.734400 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-home\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.734541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.734504 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-dshm\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.734631 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.734583 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-model-cache\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.734631 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.734614 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca463a2-02b8-46a2-8a66-fd398b50eafd-tls-certs\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.734762 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.734736 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-tmp-dir\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.775659 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.775616 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-34b13c\"/\"distinct-2-w1-34b13c-kserve-self-signed-certs\"" Nov 03 13:23:17.778343 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.778316 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg"] Nov 03 13:23:17.836166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.836135 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-tmp-dir\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.836340 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.836234 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlw5s\" (UniqueName: \"kubernetes.io/projected/cca463a2-02b8-46a2-8a66-fd398b50eafd-kube-api-access-nlw5s\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.836340 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.836291 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-home\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.836340 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.836320 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-dshm\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.836487 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.836353 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-model-cache\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.836487 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.836381 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca463a2-02b8-46a2-8a66-fd398b50eafd-tls-certs\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.836652 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.836632 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-tmp-dir\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.836720 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.836680 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-home\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.836804 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.836757 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-model-cache\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.838524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.838504 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-dshm\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.838915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.838899 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca463a2-02b8-46a2-8a66-fd398b50eafd-tls-certs\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.890545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.890508 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlw5s\" (UniqueName: \"kubernetes.io/projected/cca463a2-02b8-46a2-8a66-fd398b50eafd-kube-api-access-nlw5s\") pod \"distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:17.999132 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:17.999041 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:18.366530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:18.366492 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg"] Nov 03 13:23:18.367763 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:23:18.367736 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcca463a2_02b8_46a2_8a66_fd398b50eafd.slice/crio-d0b33e46328e27247bd23c64bb3f35c835d5cdf281059a6785cb301e3ad9092e WatchSource:0}: Error finding container d0b33e46328e27247bd23c64bb3f35c835d5cdf281059a6785cb301e3ad9092e: Status 404 returned error can't find the container with id d0b33e46328e27247bd23c64bb3f35c835d5cdf281059a6785cb301e3ad9092e Nov 03 13:23:19.184108 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:19.184069 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" event={"ID":"cca463a2-02b8-46a2-8a66-fd398b50eafd","Type":"ContainerStarted","Data":"d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872"} Nov 03 13:23:19.184602 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:19.184120 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" event={"ID":"cca463a2-02b8-46a2-8a66-fd398b50eafd","Type":"ContainerStarted","Data":"d0b33e46328e27247bd23c64bb3f35c835d5cdf281059a6785cb301e3ad9092e"} Nov 03 13:23:19.302073 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:19.302013 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" podStartSLOduration=2.301999537 podStartE2EDuration="2.301999537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:19.299724451 +0000 UTC m=+1130.613696310" watchObservedRunningTime="2026-11-03 13:23:19.301999537 +0000 UTC m=+1130.615971397" Nov 03 13:23:19.999619 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:19.999571 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:27.999895 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:27.999859 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:28.012736 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.012710 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:28.192122 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.192088 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq"] Nov 03 13:23:28.196059 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.196039 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.249008 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.248975 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"distinct-w3-5bb463-kserve-self-signed-certs\"" Nov 03 13:23:28.252302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.252220 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq"] Nov 03 13:23:28.254003 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.253975 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:23:28.370056 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.370021 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293e19ad-0cde-46c2-8039-49927287f258-tls-certs\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.370236 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.370095 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-home\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.370236 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.370201 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-model-cache\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.370236 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.370229 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5627f\" (UniqueName: \"kubernetes.io/projected/293e19ad-0cde-46c2-8039-49927287f258-kube-api-access-5627f\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.370413 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.370388 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-dshm\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.370474 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.370461 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-tmp-dir\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.471621 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.471580 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-tmp-dir\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.471781 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.471664 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293e19ad-0cde-46c2-8039-49927287f258-tls-certs\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.471781 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.471729 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-home\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.471916 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.471835 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-model-cache\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.471916 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.471872 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5627f\" (UniqueName: \"kubernetes.io/projected/293e19ad-0cde-46c2-8039-49927287f258-kube-api-access-5627f\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.472025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.471960 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-dshm\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.472082 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.472062 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-tmp-dir\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.472145 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.472116 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-home\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.472207 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.472184 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-model-cache\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.474062 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.474035 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-dshm\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.474201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.474185 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293e19ad-0cde-46c2-8039-49927287f258-tls-certs\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.529101 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.529027 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5627f\" (UniqueName: \"kubernetes.io/projected/293e19ad-0cde-46c2-8039-49927287f258-kube-api-access-5627f\") pod \"distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:28.808693 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:28.808603 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:29.178696 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:29.178673 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq"] Nov 03 13:23:29.180303 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:23:29.180276 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod293e19ad_0cde_46c2_8039_49927287f258.slice/crio-6b994fd98906580a6a58b8db7819d7c4f0867ca6ee29edcf71b3d8a0313cf150 WatchSource:0}: Error finding container 6b994fd98906580a6a58b8db7819d7c4f0867ca6ee29edcf71b3d8a0313cf150: Status 404 returned error can't find the container with id 6b994fd98906580a6a58b8db7819d7c4f0867ca6ee29edcf71b3d8a0313cf150 Nov 03 13:23:29.246149 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:29.246114 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" event={"ID":"293e19ad-0cde-46c2-8039-49927287f258","Type":"ContainerStarted","Data":"358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033"} Nov 03 13:23:29.246276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:29.246160 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" event={"ID":"293e19ad-0cde-46c2-8039-49927287f258","Type":"ContainerStarted","Data":"6b994fd98906580a6a58b8db7819d7c4f0867ca6ee29edcf71b3d8a0313cf150"} Nov 03 13:23:29.357352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:29.357295 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" podStartSLOduration=1.357279485 podStartE2EDuration="1.357279485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:29.356180784 +0000 UTC m=+1140.670152645" watchObservedRunningTime="2026-11-03 13:23:29.357279485 +0000 UTC m=+1140.671251344" Nov 03 13:23:29.809030 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:29.808997 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:34.873296 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.873265 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt"] Nov 03 13:23:34.877214 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.877197 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.926933 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.926875 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3c5dbc-openshift-default-dockercfg-hfbq4\"" Nov 03 13:23:34.942872 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.942841 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.943050 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.942881 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.943050 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.942901 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrhzn\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-kube-api-access-xrhzn\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.943050 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.943022 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.943558 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.943529 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.943742 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.943725 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.943892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.943876 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.944053 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.944039 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.944195 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.944181 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce84d7a-3f43-41b5-947e-57d90b701707-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:34.976506 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:34.976469 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt"] Nov 03 13:23:35.044951 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.044898 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.044951 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.044961 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045177 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045128 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrhzn\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-kube-api-access-xrhzn\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045215 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045174 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045260 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045242 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045312 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045297 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045349 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045335 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045392 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045373 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045452 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045440 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045490 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045483 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce84d7a-3f43-41b5-947e-57d90b701707-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045645 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045831 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045723 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.045906 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.045883 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.046255 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.046236 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce84d7a-3f43-41b5-947e-57d90b701707-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.047510 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.047489 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.047697 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.047678 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.099341 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.099302 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.099633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.099611 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrhzn\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-kube-api-access-xrhzn\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.190504 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.190468 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:35.607003 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.606973 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt"] Nov 03 13:23:35.607962 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:23:35.607933 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce84d7a_3f43_41b5_947e_57d90b701707.slice/crio-da63e20d2afea1cc9d996b7088c4e9a2ef884280d1498ab0c68e359b4bbc0d75 WatchSource:0}: Error finding container da63e20d2afea1cc9d996b7088c4e9a2ef884280d1498ab0c68e359b4bbc0d75: Status 404 returned error can't find the container with id da63e20d2afea1cc9d996b7088c4e9a2ef884280d1498ab0c68e359b4bbc0d75 Nov 03 13:23:35.610516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.610481 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:23:35.610593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.610546 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:23:35.610593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:35.610579 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:23:36.286880 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:36.286837 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" event={"ID":"2ce84d7a-3f43-41b5-947e-57d90b701707","Type":"ContainerStarted","Data":"f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83"} Nov 03 13:23:36.287349 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:36.286900 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" event={"ID":"2ce84d7a-3f43-41b5-947e-57d90b701707","Type":"ContainerStarted","Data":"da63e20d2afea1cc9d996b7088c4e9a2ef884280d1498ab0c68e359b4bbc0d75"} Nov 03 13:23:36.420032 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:36.419974 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" podStartSLOduration=2.419960271 podStartE2EDuration="2.419960271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:36.41790026 +0000 UTC m=+1147.731872120" watchObservedRunningTime="2026-11-03 13:23:36.419960271 +0000 UTC m=+1147.733932131" Nov 03 13:23:37.190790 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:37.190745 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:37.191949 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:37.191928 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:23:37.192018 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:37.191974 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 13:23:38.191144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:38.191103 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:23:38.191621 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:38.191152 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 13:23:38.808944 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:38.808908 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:38.821583 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:38.821552 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:39.191836 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:39.191801 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:23:39.192221 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:39.191865 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 13:23:39.313977 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:39.313945 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:23:40.191787 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:40.191750 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:23:40.191960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:40.191819 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 13:23:41.191166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:41.191131 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:23:41.191363 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:41.191189 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 13:23:42.191875 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:42.191838 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 03 13:23:42.192276 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:42.191897 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 03 13:23:43.196864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:43.196831 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:43.197275 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:43.197211 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:43.198058 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:43.198037 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:23:46.890114 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:23:46.890080 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:23:48.775326 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.775290 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5"] Nov 03 13:23:48.779044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.779024 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:48.879071 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.879035 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5"] Nov 03 13:23:48.905800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.905754 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvh6d\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-kube-api-access-dvh6d\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:48.905981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.905855 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:48.905981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.905948 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2441ba-5c60-47a7-bad3-749229ec5296-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:48.906110 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.906049 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:48.906110 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.906084 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:48.906211 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.906111 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:48.906211 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.906135 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:48.906298 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.906263 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:48.906517 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:48.906489 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.007857 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.007813 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2441ba-5c60-47a7-bad3-749229ec5296-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008069 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.007898 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008069 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.007922 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008069 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.007940 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008069 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.007978 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008069 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.008033 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008362 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.008098 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008362 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.008131 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvh6d\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-kube-api-access-dvh6d\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008362 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.008183 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.008379 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.008439 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008732 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.008696 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2441ba-5c60-47a7-bad3-749229ec5296-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.008725 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.008799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.008744 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.010460 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.010435 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.010583 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.010565 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.062535 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.062465 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.062749 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.062724 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvh6d\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-kube-api-access-dvh6d\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.092639 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.092609 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:49.514425 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.514396 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5"] Nov 03 13:23:49.514940 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:23:49.514901 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2441ba_5c60_47a7_bad3_749229ec5296.slice/crio-aa7ec9b4cd541452fa81c04e4d97b3a382d621259a2af56fb47e56ee4b6976e5 WatchSource:0}: Error finding container aa7ec9b4cd541452fa81c04e4d97b3a382d621259a2af56fb47e56ee4b6976e5: Status 404 returned error can't find the container with id aa7ec9b4cd541452fa81c04e4d97b3a382d621259a2af56fb47e56ee4b6976e5 Nov 03 13:23:49.517252 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.517214 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:23:49.517352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.517282 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:23:49.517352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:49.517310 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:23:50.365080 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:50.365041 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" event={"ID":"ce2441ba-5c60-47a7-bad3-749229ec5296","Type":"ContainerStarted","Data":"28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09"} Nov 03 13:23:50.365497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:50.365100 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" event={"ID":"ce2441ba-5c60-47a7-bad3-749229ec5296","Type":"ContainerStarted","Data":"aa7ec9b4cd541452fa81c04e4d97b3a382d621259a2af56fb47e56ee4b6976e5"} Nov 03 13:23:50.524227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:50.523687 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" podStartSLOduration=2.523668163 podStartE2EDuration="2.523668163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:50.52283098 +0000 UTC m=+1161.836802837" watchObservedRunningTime="2026-11-03 13:23:50.523668163 +0000 UTC m=+1161.837640026" Nov 03 13:23:51.093501 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:51.093468 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:51.095255 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:51.095226 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:23:51.095390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:51.095279 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 13:23:52.093250 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:52.093210 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:23:52.093650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:52.093279 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 13:23:53.093720 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:53.093679 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:23:53.094142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:53.093743 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 13:23:54.093356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:54.093319 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:23:54.093542 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:54.093385 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 13:23:55.093840 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:55.093801 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:23:55.094233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:55.093891 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 13:23:56.093473 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:56.093433 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:23:56.093649 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:56.093504 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 03 13:23:57.098185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:57.098149 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:57.098670 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:57.098547 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:23:57.099135 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:23:57.099114 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:24:16.891264 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:24:16.891225 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:24:20.249762 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.249709 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf"] Nov 03 13:24:20.279205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.279172 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.386219 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.385474 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf"] Nov 03 13:24:20.512098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.512009 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.512248 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.512117 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.512248 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.512158 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.512248 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.512211 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.512363 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.512247 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd2zh\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-kube-api-access-fd2zh\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.512397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.512380 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.512430 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.512419 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.512511 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.512489 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.512565 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.512554 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613220 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613176 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2zh\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-kube-api-access-fd2zh\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613270 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613295 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613330 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613400 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613456 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613512 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613552 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613946 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613750 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.613946 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613831 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.614075 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.613951 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.614127 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.614096 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.614183 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.614153 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.614244 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.614153 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.615810 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.615790 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.615945 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.615926 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.669404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.669370 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2zh\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-kube-api-access-fd2zh\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.669588 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.669409 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:20.915595 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:20.915562 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:21.339313 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:21.339284 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf"] Nov 03 13:24:21.340580 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:24:21.340553 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaccd7db_1711_4a3f_9ef5_cf0624d88b22.slice/crio-35f74d2ef446cbcc1ecddcb7660b9877e643c4f8774cbdbaecc7c7fc6c238d9a WatchSource:0}: Error finding container 35f74d2ef446cbcc1ecddcb7660b9877e643c4f8774cbdbaecc7c7fc6c238d9a: Status 404 returned error can't find the container with id 35f74d2ef446cbcc1ecddcb7660b9877e643c4f8774cbdbaecc7c7fc6c238d9a Nov 03 13:24:21.342928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:21.342894 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:21.343003 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:21.342959 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:21.343003 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:21.342989 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:21.555484 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:21.555452 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" event={"ID":"aaccd7db-1711-4a3f-9ef5-cf0624d88b22","Type":"ContainerStarted","Data":"3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941"} Nov 03 13:24:21.555645 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:21.555507 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" event={"ID":"aaccd7db-1711-4a3f-9ef5-cf0624d88b22","Type":"ContainerStarted","Data":"35f74d2ef446cbcc1ecddcb7660b9877e643c4f8774cbdbaecc7c7fc6c238d9a"} Nov 03 13:24:21.673270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:21.673190 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" podStartSLOduration=1.673170759 podStartE2EDuration="1.673170759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:21.671105384 +0000 UTC m=+1192.985077243" watchObservedRunningTime="2026-11-03 13:24:21.673170759 +0000 UTC m=+1192.987142620" Nov 03 13:24:21.915955 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:21.915920 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:22.918125 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:22.918084 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:24:22.918544 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:22.918141 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 13:24:23.916847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:23.916812 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:24:23.917033 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:23.916874 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 13:24:24.916687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:24.916649 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:24:24.917124 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:24.916703 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 13:24:25.916529 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:25.916494 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:24:25.916726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:25.916550 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 13:24:26.916222 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:26.916189 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:24:26.916406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:26.916240 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 13:24:27.916837 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:27.916797 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 03 13:24:27.917287 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:27.916867 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 03 13:24:28.920858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:28.920830 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:28.921245 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:28.921189 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:28.921938 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:28.921916 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:24:29.008198 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:29.008164 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:24:29.009456 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:29.009432 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:24:40.734022 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.733750 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv"] Nov 03 13:24:40.743114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.743081 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:40.796795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.794860 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42c707-openshift-default-dockercfg-7t7mv\"" Nov 03 13:24:40.876421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.876378 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:40.876742 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.876715 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:40.876939 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.876918 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xngj6\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-kube-api-access-xngj6\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:40.890794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.877098 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363b2c47-22f2-4305-93ab-83b8680d1e01-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:40.891040 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.891015 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:40.891197 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.891179 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:40.914797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.891320 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:40.915116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.915084 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:40.915295 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:40.915272 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.020852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.020244 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.020852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.020353 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xngj6\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-kube-api-access-xngj6\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.020852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.020463 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363b2c47-22f2-4305-93ab-83b8680d1e01-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.020852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.020522 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.020852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.020551 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.020852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.020611 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.020852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.020659 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.020852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.020701 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.020852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.020756 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.040794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.021825 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.104795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.085740 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv"] Nov 03 13:24:41.104795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.086166 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.104795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.086539 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363b2c47-22f2-4305-93ab-83b8680d1e01-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.235797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.235131 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.235797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.235646 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.235797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.235789 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.239787 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.239463 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xngj6\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-kube-api-access-xngj6\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.240247 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.240223 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.245536 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.245511 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.372841 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.371986 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:41.805925 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:41.805873 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:24:42.028186 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:42.028141 4541 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:24:42.303796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:42.300293 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" podUID="e961fb1a-50db-4f9b-9a17-b690246f59a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:24:42.373795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:42.373483 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv"] Nov 03 13:24:42.458329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:42.457566 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:42.458329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:42.457665 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:42.458329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:42.457712 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:24:42.828973 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:42.828936 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" event={"ID":"363b2c47-22f2-4305-93ab-83b8680d1e01","Type":"ContainerStarted","Data":"85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262"} Nov 03 13:24:42.829332 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:42.829026 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" event={"ID":"363b2c47-22f2-4305-93ab-83b8680d1e01","Type":"ContainerStarted","Data":"0ab98afbef1f577f9ebad46dff1b2679c4270401bced1177a4413b50b7d69f94"} Nov 03 13:24:42.992796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:42.992581 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podStartSLOduration=2.992559826 podStartE2EDuration="2.992559826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:42.975472305 +0000 UTC m=+1214.289444166" watchObservedRunningTime="2026-11-03 13:24:42.992559826 +0000 UTC m=+1214.306531689" Nov 03 13:24:43.374849 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:43.374449 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:43.374849 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:43.374608 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:24:43.374849 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:43.374659 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 13:24:44.373473 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:44.373424 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:24:44.373936 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:44.373503 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 13:24:45.373136 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:45.373099 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:24:45.373319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:45.373153 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 13:24:46.373236 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:46.373194 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:24:46.373654 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:46.373249 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 13:24:46.889666 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:24:46.889627 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:24:47.373106 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:47.373065 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:24:47.373302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:47.373141 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 13:24:48.372681 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:48.372640 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:24:48.372890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:48.372719 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 13:24:49.373608 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:49.373549 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 13:24:49.373987 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:49.373632 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 13:24:50.377207 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:50.377177 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:50.377651 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:50.377556 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:24:50.378277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:24:50.378252 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:25:05.682738 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.682696 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt"] Nov 03 13:25:05.687147 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.687125 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.792256 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.792221 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt"] Nov 03 13:25:05.812390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.811010 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.812390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.811126 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.812390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.811248 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.812390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.811283 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.812390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.811330 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.812390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.811349 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rvp\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-kube-api-access-w2rvp\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.812390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.811379 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.812390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.811429 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.812390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.811455 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912171 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912331 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912378 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912417 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912460 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912588 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912514 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912588 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912546 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rvp\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-kube-api-access-w2rvp\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912592 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912659 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912808 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912724 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912808 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912661 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.912925 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.912821 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.913094 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.913062 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.913289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.913267 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.914882 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.914861 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.915013 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.914998 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.974070 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.973999 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:05.990813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:05.990784 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rvp\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-kube-api-access-w2rvp\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:06.001078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:06.001039 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:06.423895 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:06.423855 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt"] Nov 03 13:25:06.457099 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:25:06.457070 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37545439_ddd2_40d6_bf2f_a2b8c926ad5e.slice/crio-5a447d83036e16e4afb941c1f19a661241c34c0054889b86a10e727a216847a9 WatchSource:0}: Error finding container 5a447d83036e16e4afb941c1f19a661241c34c0054889b86a10e727a216847a9: Status 404 returned error can't find the container with id 5a447d83036e16e4afb941c1f19a661241c34c0054889b86a10e727a216847a9 Nov 03 13:25:06.459506 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:06.459470 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:06.459607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:06.459559 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:06.459607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:06.459589 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:07.006254 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:07.006215 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" event={"ID":"37545439-ddd2-40d6-bf2f-a2b8c926ad5e","Type":"ContainerStarted","Data":"085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc"} Nov 03 13:25:07.006759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:07.006276 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" event={"ID":"37545439-ddd2-40d6-bf2f-a2b8c926ad5e","Type":"ContainerStarted","Data":"5a447d83036e16e4afb941c1f19a661241c34c0054889b86a10e727a216847a9"} Nov 03 13:25:07.125642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:07.125584 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" podStartSLOduration=2.125568829 podStartE2EDuration="2.125568829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:07.124217433 +0000 UTC m=+1238.438189293" watchObservedRunningTime="2026-11-03 13:25:07.125568829 +0000 UTC m=+1238.439540689" Nov 03 13:25:08.001169 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:08.001137 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:08.002700 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:08.002673 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:08.002842 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:08.002722 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 13:25:09.002284 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:09.002247 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:09.002703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:09.002311 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 13:25:10.002031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:10.001988 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:10.002237 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:10.002060 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 13:25:11.001577 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:11.001547 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:11.001984 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:11.001609 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 13:25:12.001617 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:12.001584 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:12.002023 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:12.001636 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 13:25:13.001868 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:13.001833 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:25:13.002259 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:13.001890 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 13:25:14.005780 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:14.005735 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:14.006246 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:14.006188 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:14.006887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:14.006867 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:25:16.889968 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:25:16.889933 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:25:20.698920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:20.698863 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:20.804911 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:20.804860 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:20.901795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:20.897853 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" podUID="aafbdc79-6bde-48a0-853c-8a92f80481eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:21.073851 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:21.072974 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:21.252949 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:21.252901 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" podUID="cca463a2-02b8-46a2-8a66-fd398b50eafd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:21.283765 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:21.283724 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" podUID="e961fb1a-50db-4f9b-9a17-b690246f59a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 03 13:25:34.663731 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.663695 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c"] Nov 03 13:25:34.667694 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.667675 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.843941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.843905 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.844094 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.843949 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.844185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.844157 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.844240 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.844216 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.844289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.844254 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.844402 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.844390 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.844458 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.844415 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk557\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-kube-api-access-mk557\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.844556 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.844536 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.844618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.844593 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.856653 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.856620 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c"] Nov 03 13:25:34.870832 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.870798 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r"] Nov 03 13:25:34.874563 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.874547 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:34.924236 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.924147 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5bb463-openshift-default-dockercfg-jldlg\"" Nov 03 13:25:34.932007 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.931973 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r"] Nov 03 13:25:34.946247 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.946215 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.946424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.946280 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.946424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.946316 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.946424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.946391 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.946424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.946419 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk557\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-kube-api-access-mk557\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.946646 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.946504 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.946646 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.946600 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.946751 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.946666 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.946751 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.946723 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.947373 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.947345 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.947699 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.947670 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.947813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.947678 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.948040 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.948018 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.948167 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.948143 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.948895 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.948861 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:34.949007 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:34.948991 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:35.000631 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.000595 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:35.024575 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.024545 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk557\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-kube-api-access-mk557\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:35.047859 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.047827 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.048000 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.047959 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.048000 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.047988 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.048073 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.048013 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlwn\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-kube-api-access-pwlwn\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.048073 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.048040 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.048253 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.048236 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.048299 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.048291 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.048372 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.048358 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.048406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.048383 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.149977 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150017 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150044 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150156 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150217 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150305 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150332 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150368 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlwn\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-kube-api-access-pwlwn\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150406 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150410 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150731 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150499 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150731 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150678 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150731 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150717 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.150941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.150891 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.152504 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.152475 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.152753 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.152735 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.205288 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.205206 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.205436 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.205307 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlwn\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-kube-api-access-pwlwn\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.280765 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.280728 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:35.488741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.488658 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:35.706744 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.706714 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c"] Nov 03 13:25:35.707929 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:25:35.707901 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86bbf92d_72fe_49d6_9e1b_6265bdf1b809.slice/crio-8b14ddf4dd2b708a32419d81eab7c7b83b4d7ac0f248160d5bd4d615ade173c8 WatchSource:0}: Error finding container 8b14ddf4dd2b708a32419d81eab7c7b83b4d7ac0f248160d5bd4d615ade173c8: Status 404 returned error can't find the container with id 8b14ddf4dd2b708a32419d81eab7c7b83b4d7ac0f248160d5bd4d615ade173c8 Nov 03 13:25:35.710277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.710241 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:35.710357 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.710312 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:35.710357 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.710340 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:35.862871 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.862841 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r"] Nov 03 13:25:35.865054 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:25:35.865023 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4e92b6_c318_43f8_8a8e_e7c421895a49.slice/crio-85e1c9f27d1409608e38b78dcd5c01f6e3909db7fe877de994d6005d549c2ed8 WatchSource:0}: Error finding container 85e1c9f27d1409608e38b78dcd5c01f6e3909db7fe877de994d6005d549c2ed8: Status 404 returned error can't find the container with id 85e1c9f27d1409608e38b78dcd5c01f6e3909db7fe877de994d6005d549c2ed8 Nov 03 13:25:35.867917 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.867881 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:35.868023 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.867965 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:35.868023 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:35.868005 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:36.366358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:36.366316 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" event={"ID":"86bbf92d-72fe-49d6-9e1b-6265bdf1b809","Type":"ContainerStarted","Data":"e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8"} Nov 03 13:25:36.366523 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:36.366385 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" event={"ID":"86bbf92d-72fe-49d6-9e1b-6265bdf1b809","Type":"ContainerStarted","Data":"8b14ddf4dd2b708a32419d81eab7c7b83b4d7ac0f248160d5bd4d615ade173c8"} Nov 03 13:25:36.368043 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:36.368013 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" event={"ID":"3c4e92b6-c318-43f8-8a8e-e7c421895a49","Type":"ContainerStarted","Data":"ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23"} Nov 03 13:25:36.368169 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:36.368058 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" event={"ID":"3c4e92b6-c318-43f8-8a8e-e7c421895a49","Type":"ContainerStarted","Data":"85e1c9f27d1409608e38b78dcd5c01f6e3909db7fe877de994d6005d549c2ed8"} Nov 03 13:25:36.479706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:36.479630 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" podStartSLOduration=2.4796132 podStartE2EDuration="2.4796132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:36.477024301 +0000 UTC m=+1267.790996162" watchObservedRunningTime="2026-11-03 13:25:36.4796132 +0000 UTC m=+1267.793585061" Nov 03 13:25:36.489146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:36.489113 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:36.490560 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:36.490537 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:25:36.490622 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:36.490587 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 13:25:37.287796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:37.281466 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:37.290424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:37.289914 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:25:37.290424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:37.289981 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 13:25:37.489374 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:37.489341 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:25:37.489560 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:37.489397 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 13:25:38.284121 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:38.284082 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:25:38.284418 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:38.284388 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 13:25:38.489831 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:38.489801 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:25:38.490296 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:38.490267 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 13:25:39.283701 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:39.283663 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:25:39.284446 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:39.284410 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 13:25:39.489252 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:39.489214 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:25:39.489541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:39.489504 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 13:25:40.284798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:40.282184 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:25:40.284798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:40.282264 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 13:25:40.489954 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:40.489911 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:25:40.490143 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:40.489979 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 13:25:41.284410 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:41.284378 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:25:41.284686 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:41.284658 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 13:25:41.489943 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:41.489903 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:25:41.490321 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:41.489962 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 13:25:42.282141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:42.282103 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:25:42.282400 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:42.282373 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 03 13:25:42.489922 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:42.489876 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:25:42.490097 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:42.489950 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 13:25:43.297811 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:43.297755 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:43.298533 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:43.298493 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:43.299535 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:43.299515 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:25:43.466510 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:43.466441 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" podStartSLOduration=9.466419025 podStartE2EDuration="9.466419025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:36.595947073 +0000 UTC m=+1267.909918934" watchObservedRunningTime="2026-11-03 13:25:43.466419025 +0000 UTC m=+1274.780390886" Nov 03 13:25:43.529758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:43.529720 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:43.530273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:43.530195 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:43.531350 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:43.531330 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:25:46.890576 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:25:46.890535 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:25:50.035817 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.026439 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz"] Nov 03 13:25:50.055269 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.054079 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.130020 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.129984 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.130323 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.130299 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9046e25c-55b4-4746-a618-8e8b48a69641-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.130475 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.130457 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r625k\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-kube-api-access-r625k\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.130623 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.130604 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.130814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.130797 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.130957 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.130941 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.131145 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.131128 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.131270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.131255 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9046e25c-55b4-4746-a618-8e8b48a69641-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.131426 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.131407 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.171798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.146524 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz"] Nov 03 13:25:50.299854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.274650 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.300177 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.300142 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9046e25c-55b4-4746-a618-8e8b48a69641-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.313727 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.313683 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r625k\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-kube-api-access-r625k\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.314025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.314002 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.314203 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.314185 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.331904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.331866 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.332236 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.332219 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.335845 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.335825 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9046e25c-55b4-4746-a618-8e8b48a69641-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.356792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.344907 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.356792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.316807 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9046e25c-55b4-4746-a618-8e8b48a69641-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.356792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.333720 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.356792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.317089 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.356792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.317863 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.356792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.333474 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.735797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.735684 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.739792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.738423 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9046e25c-55b4-4746-a618-8e8b48a69641-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.739792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.738740 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.741130 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.741087 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 13:25:50.781803 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.771081 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:50.820981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.773050 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" podUID="824d587c-60d2-4cf2-9c31-37f8baf60308" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:50.831839 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.831806 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r625k\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-kube-api-access-r625k\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:50.902922 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:50.902864 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" podUID="aafbdc79-6bde-48a0-853c-8a92f80481eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.022847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.019005 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:51.056538 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.056487 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 13:25:51.295968 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.295847 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" podUID="e961fb1a-50db-4f9b-9a17-b690246f59a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.319948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.319898 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" podUID="293e19ad-0cde-46c2-8039-49927287f258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.440794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.440566 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:25:51.440794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.440633 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.475796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.475162 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" podUID="d6edf898-5a79-4539-a61a-ea6a379c98d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.747795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.746861 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.747795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.747613 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:25:51.784015 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.776862 4541 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.950795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.826948 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 13:25:51.950795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.827384 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:25:51.950795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.844391 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Nov 03 13:25:51.950795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.868721 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" podUID="824d587c-60d2-4cf2-9c31-37f8baf60308" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.950795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.869185 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:25:51.950795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.913904 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" podUID="aafbdc79-6bde-48a0-853c-8a92f80481eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.950795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:51.914612 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:25:52.040796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.014902 4541 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:52.053797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.041640 4541 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" podUID="e961fb1a-50db-4f9b-9a17-b690246f59a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:52.065904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.065845 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:52.068409 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.066994 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:25:52.189520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.189490 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:25:52.257976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.257934 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:25:52.278396 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.275902 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 13:25:52.302121 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.299912 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:25:52.303038 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.303015 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 13:25:52.334917 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.334878 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" podUID="293e19ad-0cde-46c2-8039-49927287f258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:52.335818 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.335798 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:25:52.370302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.370266 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz"] Nov 03 13:25:52.431801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.431755 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 13:25:52.475090 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.475048 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:52.475416 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.475392 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:52.475585 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:52.475564 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:25:53.145859 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:53.145820 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" event={"ID":"9046e25c-55b4-4746-a618-8e8b48a69641","Type":"ContainerStarted","Data":"a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03"} Nov 03 13:25:53.146306 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:53.145885 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" event={"ID":"9046e25c-55b4-4746-a618-8e8b48a69641","Type":"ContainerStarted","Data":"6dce0a26aed16f485038fa784f075badce6aa36a1f8ce4a5f489b13ee0c5a4e4"} Nov 03 13:25:53.279185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:53.279131 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" podStartSLOduration=4.279116266 podStartE2EDuration="4.279116266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:53.277296748 +0000 UTC m=+1284.591268608" watchObservedRunningTime="2026-11-03 13:25:53.279116266 +0000 UTC m=+1284.593088127" Nov 03 13:25:54.019835 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:54.019462 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:25:54.020901 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:54.020657 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 13:25:54.020901 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:54.020709 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 13:25:55.019969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:55.019934 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 13:25:55.020370 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:55.019988 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 13:25:56.020026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:56.019992 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 13:25:56.020501 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:56.020172 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 13:25:57.020646 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:57.020603 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 13:25:57.021128 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:57.020664 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 13:25:58.020571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:58.020536 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 13:25:58.020724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:58.020591 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 13:25:59.019801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:59.019757 4541 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 03 13:25:59.020009 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:25:59.019832 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 03 13:26:00.187793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:00.184473 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:26:00.187793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:00.185259 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:26:00.187793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:00.186275 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:26:11.492177 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.492093 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd"] Nov 03 13:26:11.497546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.497526 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.623722 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.623689 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.623911 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.623754 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.623911 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.623890 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.624025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.623934 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.624025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.623959 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr2pv\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-kube-api-access-rr2pv\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.624138 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.624083 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.624191 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.624141 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.624227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.624205 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.624294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.624280 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.695103 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.695069 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd"] Nov 03 13:26:11.725498 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725466 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.725655 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725532 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.725655 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725613 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.725795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725669 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.725795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725717 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.725795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725751 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr2pv\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-kube-api-access-rr2pv\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.725969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725806 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.725969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725847 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.725969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725881 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.725969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.725930 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.726179 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.726106 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.726353 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.726327 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.726459 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.726429 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.726608 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.726582 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.728375 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.728349 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.728529 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.728515 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.783403 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.783339 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr2pv\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-kube-api-access-rr2pv\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.783403 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.783343 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:11.810429 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:11.810401 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:12.226116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:12.226090 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd"] Nov 03 13:26:12.227863 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:26:12.227818 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cafd0c_7aa9_498d_8f7b_ca52431af32d.slice/crio-83739b8053b9c10dce6fdb840fa8e2e702f44835418abf180e8e5082dedacfbf WatchSource:0}: Error finding container 83739b8053b9c10dce6fdb840fa8e2e702f44835418abf180e8e5082dedacfbf: Status 404 returned error can't find the container with id 83739b8053b9c10dce6fdb840fa8e2e702f44835418abf180e8e5082dedacfbf Nov 03 13:26:12.230485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:12.230439 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:26:12.230583 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:12.230532 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:26:12.230627 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:12.230580 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:26:12.302572 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:12.302536 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" event={"ID":"16cafd0c-7aa9-498d-8f7b-ca52431af32d","Type":"ContainerStarted","Data":"83739b8053b9c10dce6fdb840fa8e2e702f44835418abf180e8e5082dedacfbf"} Nov 03 13:26:13.364790 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:13.354580 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" event={"ID":"16cafd0c-7aa9-498d-8f7b-ca52431af32d","Type":"ContainerStarted","Data":"c051e2e60cdb4b5954a98215304609ed02d88caa348b209ad41cd7ebd013afde"} Nov 03 13:26:13.544798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:13.541200 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" podStartSLOduration=2.541177156 podStartE2EDuration="2.541177156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:13.528759941 +0000 UTC m=+1304.842731804" watchObservedRunningTime="2026-11-03 13:26:13.541177156 +0000 UTC m=+1304.855149017" Nov 03 13:26:13.813796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:13.813731 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:13.814011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:13.813889 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:26:13.814011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:13.813950 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 13:26:14.207989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.207945 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" podUID="824d587c-60d2-4cf2-9c31-37f8baf60308" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.243905 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.243854 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" podUID="cca463a2-02b8-46a2-8a66-fd398b50eafd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.275822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.273864 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.282795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.279385 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" podUID="aafbdc79-6bde-48a0-853c-8a92f80481eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Nov 03 13:26:14.283054 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.283022 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5" podUID="e961fb1a-50db-4f9b-9a17-b690246f59a7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.312913 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.312860 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.313489 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.313458 4541 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.314137 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.314110 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.440334 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.437842 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" podUID="d6edf898-5a79-4539-a61a-ea6a379c98d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.440334 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.438204 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" podUID="293e19ad-0cde-46c2-8039-49927287f258" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Nov 03 13:26:14.803795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.802907 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.823795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.813248 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:26:14.823795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.813322 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 13:26:14.967909 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.967870 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:26:14.968203 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:14.968170 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:15.209750 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.208855 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" podUID="824d587c-60d2-4cf2-9c31-37f8baf60308" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:15.209750 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.209556 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:26:15.246482 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.244874 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" podUID="cca463a2-02b8-46a2-8a66-fd398b50eafd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Nov 03 13:26:15.246482 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.245258 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:26:15.284145 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.284102 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Nov 03 13:26:15.284930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.284890 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:26:15.316766 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.316724 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:15.317572 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.317548 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:26:15.420989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.420288 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 13:26:15.437792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.435835 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 13:26:15.512651 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.496966 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 13:26:15.518235 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.518206 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 13:26:15.813858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.812968 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:26:15.813858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:15.813028 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 13:26:16.810996 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:16.810963 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:26:16.811383 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:16.811024 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 13:26:16.915727 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:26:16.915684 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:26:17.811156 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:17.811121 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:26:17.811532 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:17.811179 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 13:26:18.811041 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:18.810998 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 03 13:26:18.811253 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:18.811062 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 03 13:26:20.517794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:20.517345 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:20.527860 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:20.527836 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:20.542972 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:20.538013 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:26:42.264792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:42.262877 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:26:42.264792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:26:42.262979 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:46.890762 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:26:46.890715 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:27:11.957110 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:11.957081 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt"] Nov 03 13:27:11.961069 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:11.961053 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.102573 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.102530 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.102812 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.102785 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.102902 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.102850 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.102961 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.102941 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.103000 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.102970 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341f6a95-e931-4864-9caf-975236c1d7f9-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.103000 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.102988 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj95c\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-kube-api-access-dj95c\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.103099 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.103009 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.103155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.103102 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.103155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.103139 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341f6a95-e931-4864-9caf-975236c1d7f9-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.171928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.171894 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt"] Nov 03 13:27:12.204587 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.204551 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.204767 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.204644 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.204767 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.204697 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205057 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205032 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341f6a95-e931-4864-9caf-975236c1d7f9-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205162 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205134 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205162 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205153 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj95c\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-kube-api-access-dj95c\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205310 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205201 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205310 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205230 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205310 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205270 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341f6a95-e931-4864-9caf-975236c1d7f9-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205462 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205333 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205926 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205652 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205926 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205730 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341f6a95-e931-4864-9caf-975236c1d7f9-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.205926 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205857 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.206167 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.205961 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.207541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.207474 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.208162 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.208134 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341f6a95-e931-4864-9caf-975236c1d7f9-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.258828 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.258796 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.259039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.259011 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj95c\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-kube-api-access-dj95c\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.274766 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.274741 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:12.695141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.695105 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt"] Nov 03 13:27:12.729590 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:27:12.729561 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341f6a95_e931_4864_9caf_975236c1d7f9.slice/crio-b4b0b9b313f04192e48bb4a053f31fdf49d7b6707b4d30d268cbe009174c6e65 WatchSource:0}: Error finding container b4b0b9b313f04192e48bb4a053f31fdf49d7b6707b4d30d268cbe009174c6e65: Status 404 returned error can't find the container with id b4b0b9b313f04192e48bb4a053f31fdf49d7b6707b4d30d268cbe009174c6e65 Nov 03 13:27:12.731974 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.731940 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:27:12.732046 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.732029 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:27:12.732082 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.732058 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:27:12.918031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.917994 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" event={"ID":"341f6a95-e931-4864-9caf-975236c1d7f9","Type":"ContainerStarted","Data":"d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f"} Nov 03 13:27:12.918193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:12.918049 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" event={"ID":"341f6a95-e931-4864-9caf-975236c1d7f9","Type":"ContainerStarted","Data":"b4b0b9b313f04192e48bb4a053f31fdf49d7b6707b4d30d268cbe009174c6e65"} Nov 03 13:27:13.030469 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:13.030361 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podStartSLOduration=2.030343651 podStartE2EDuration="2.030343651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:13.029580558 +0000 UTC m=+1364.343552420" watchObservedRunningTime="2026-11-03 13:27:13.030343651 +0000 UTC m=+1364.344315513" Nov 03 13:27:13.275688 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:13.275649 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:13.277487 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:13.277458 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:27:13.277610 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:13.277510 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 13:27:14.275847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:14.275808 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:27:14.276335 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:14.275878 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 13:27:15.275715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:15.275679 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:27:15.275936 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:15.275738 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 13:27:16.275731 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:16.275689 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:27:16.275949 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:16.275755 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 13:27:16.890335 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:27:16.890297 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:27:17.275920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:17.275873 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:27:17.276105 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:17.275934 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 13:27:18.275503 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:18.275465 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:27:18.275687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:18.275522 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 13:27:19.275663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:19.275624 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 03 13:27:19.276140 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:19.275699 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 03 13:27:20.283106 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:20.283074 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:20.283574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:20.283507 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:20.284303 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:20.284283 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:27:25.547191 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.547156 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24"] Nov 03 13:27:25.551218 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.551200 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.605246 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.605219 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 13:27:25.605246 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.605236 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 13:27:25.605389 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.605367 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ffbgv\"" Nov 03 13:27:25.649551 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.649514 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 13:27:25.653449 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.653419 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24"] Nov 03 13:27:25.693485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.693454 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.693643 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.693529 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.693691 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.693652 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k4dk\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-kube-api-access-7k4dk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.693834 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.693812 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.693913 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.693861 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.693960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.693937 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.694079 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.694060 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.795740 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.795702 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.795969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.795810 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.795969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.795870 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.795969 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.795900 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k4dk\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-kube-api-access-7k4dk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.796144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.795970 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.796144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.796007 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.796144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.796039 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.796643 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.796616 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.800749 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.798993 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.800749 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.799035 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.800749 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.799212 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.807314 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.807262 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.850394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.850370 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k4dk\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-kube-api-access-7k4dk\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.850805 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.850747 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-xwx24\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:25.864709 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:25.864690 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:26.287094 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:26.287044 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24"] Nov 03 13:27:26.289651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:27:26.289625 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdff1fd_bfbf_453d_97b2_6ee9158cf0c1.slice/crio-188fc2deb35be29fd2f4f376cc35006a2e51ebbef39f47535d440fdbb80ea884 WatchSource:0}: Error finding container 188fc2deb35be29fd2f4f376cc35006a2e51ebbef39f47535d440fdbb80ea884: Status 404 returned error can't find the container with id 188fc2deb35be29fd2f4f376cc35006a2e51ebbef39f47535d440fdbb80ea884 Nov 03 13:27:26.292297 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:26.292256 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:27:26.292407 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:26.292340 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:27:27.033305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:27.033268 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" event={"ID":"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1","Type":"ContainerStarted","Data":"25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb"} Nov 03 13:27:27.033726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:27.033326 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" event={"ID":"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1","Type":"ContainerStarted","Data":"188fc2deb35be29fd2f4f376cc35006a2e51ebbef39f47535d440fdbb80ea884"} Nov 03 13:27:27.033726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:27.033488 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:27.035069 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:27.035044 4541 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xwx24 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:27:27.035193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:27.035093 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" podUID="cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:27:27.230670 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:27.230611 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" podStartSLOduration=2.230593804 podStartE2EDuration="2.230593804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:27.192955503 +0000 UTC m=+1378.506927359" watchObservedRunningTime="2026-11-03 13:27:27.230593804 +0000 UTC m=+1378.544565668" Nov 03 13:27:28.046872 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:28.046826 4541 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-xwx24 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:27:28.047434 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:28.046904 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" podUID="cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:27:31.043334 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:27:31.043245 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:27:46.889360 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:27:46.889320 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:28:16.890657 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:28:16.890607 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:28:27.253920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:28:27.253875 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:28:27.254601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:28:27.254566 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:28:46.891573 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:28:46.891533 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:29:16.890673 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:29:16.890592 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:29:29.045023 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:29:29.044991 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:29:29.046368 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:29:29.046348 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:29:46.893345 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:29:46.893305 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:30:00.212574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.212533 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd"] Nov 03 13:30:00.216742 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.216719 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7"] Nov 03 13:30:00.216910 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.216870 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:00.220378 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.220360 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:00.308357 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.308321 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/026b4e23-e769-4652-b452-2a884975d36e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd\" (UID: \"026b4e23-e769-4652-b452-2a884975d36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:00.308541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.308410 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9ztc\" (UniqueName: \"kubernetes.io/projected/026b4e23-e769-4652-b452-2a884975d36e-kube-api-access-w9ztc\") pod \"maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd\" (UID: \"026b4e23-e769-4652-b452-2a884975d36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:00.308680 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.308662 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzp5z\" (UniqueName: \"kubernetes.io/projected/fcf391c9-d26c-408f-af2f-19a2cf151dec-kube-api-access-kzp5z\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7\" (UID: \"fcf391c9-d26c-408f-af2f-19a2cf151dec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:00.308799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.308784 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf391c9-d26c-408f-af2f-19a2cf151dec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7\" (UID: \"fcf391c9-d26c-408f-af2f-19a2cf151dec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:00.325516 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.325486 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7"] Nov 03 13:30:00.329213 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.329184 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:30:00.411505 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.411468 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd"] Nov 03 13:30:00.413933 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.413904 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf391c9-d26c-408f-af2f-19a2cf151dec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7\" (UID: \"fcf391c9-d26c-408f-af2f-19a2cf151dec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:00.414190 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.414160 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/026b4e23-e769-4652-b452-2a884975d36e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd\" (UID: \"026b4e23-e769-4652-b452-2a884975d36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:00.414288 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.414203 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9ztc\" (UniqueName: \"kubernetes.io/projected/026b4e23-e769-4652-b452-2a884975d36e-kube-api-access-w9ztc\") pod \"maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd\" (UID: \"026b4e23-e769-4652-b452-2a884975d36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:00.414478 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.414458 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzp5z\" (UniqueName: \"kubernetes.io/projected/fcf391c9-d26c-408f-af2f-19a2cf151dec-kube-api-access-kzp5z\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7\" (UID: \"fcf391c9-d26c-408f-af2f-19a2cf151dec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:00.414930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.414907 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf391c9-d26c-408f-af2f-19a2cf151dec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7\" (UID: \"fcf391c9-d26c-408f-af2f-19a2cf151dec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:00.415546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.415519 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/026b4e23-e769-4652-b452-2a884975d36e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd\" (UID: \"026b4e23-e769-4652-b452-2a884975d36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:00.470397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.470321 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9ztc\" (UniqueName: \"kubernetes.io/projected/026b4e23-e769-4652-b452-2a884975d36e-kube-api-access-w9ztc\") pod \"maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd\" (UID: \"026b4e23-e769-4652-b452-2a884975d36e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:00.470397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.470357 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzp5z\" (UniqueName: \"kubernetes.io/projected/fcf391c9-d26c-408f-af2f-19a2cf151dec-kube-api-access-kzp5z\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7\" (UID: \"fcf391c9-d26c-408f-af2f-19a2cf151dec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:00.531518 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.531485 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:00.537827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.537807 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:00.908312 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.908286 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd"] Nov 03 13:30:00.908721 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:30:00.908697 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026b4e23_e769_4652_b452_2a884975d36e.slice/crio-a7ebd76bce4fd76d3a4ab596a12b75e4a3b6ca6459b6a39fdc822513fb8544d3 WatchSource:0}: Error finding container a7ebd76bce4fd76d3a4ab596a12b75e4a3b6ca6459b6a39fdc822513fb8544d3: Status 404 returned error can't find the container with id a7ebd76bce4fd76d3a4ab596a12b75e4a3b6ca6459b6a39fdc822513fb8544d3 Nov 03 13:30:00.910630 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.910608 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:30:00.955628 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:00.955597 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7"] Nov 03 13:30:00.955755 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:30:00.955636 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf391c9_d26c_408f_af2f_19a2cf151dec.slice/crio-a29c9b7ff7bb1ca16d725dd71cfef097e7eb854d50bde77f15d7696037e73f21 WatchSource:0}: Error finding container a29c9b7ff7bb1ca16d725dd71cfef097e7eb854d50bde77f15d7696037e73f21: Status 404 returned error can't find the container with id a29c9b7ff7bb1ca16d725dd71cfef097e7eb854d50bde77f15d7696037e73f21 Nov 03 13:30:01.089150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:01.089111 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" event={"ID":"026b4e23-e769-4652-b452-2a884975d36e","Type":"ContainerStarted","Data":"a7ebd76bce4fd76d3a4ab596a12b75e4a3b6ca6459b6a39fdc822513fb8544d3"} Nov 03 13:30:01.090309 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:01.090280 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" event={"ID":"fcf391c9-d26c-408f-af2f-19a2cf151dec","Type":"ContainerStarted","Data":"a29c9b7ff7bb1ca16d725dd71cfef097e7eb854d50bde77f15d7696037e73f21"} Nov 03 13:30:06.125748 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:06.125713 4541 generic.go:356] "Generic (PLEG): container finished" podID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerID="25ea87e1884e7e6d552847dbf59e2673c88c91ae01623b790679d1d0631131da" exitCode=60 Nov 03 13:30:06.126212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:06.125821 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" event={"ID":"fcf391c9-d26c-408f-af2f-19a2cf151dec","Type":"ContainerDied","Data":"25ea87e1884e7e6d552847dbf59e2673c88c91ae01623b790679d1d0631131da"} Nov 03 13:30:06.126212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:06.126099 4541 scope.go:122] "RemoveContainer" containerID="25ea87e1884e7e6d552847dbf59e2673c88c91ae01623b790679d1d0631131da" Nov 03 13:30:06.127607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:06.127581 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" event={"ID":"026b4e23-e769-4652-b452-2a884975d36e","Type":"ContainerStarted","Data":"5aa8476ab04f3e5bbd4d8f2184840467aaac442918b493cbd8a15b7f7233dbee"} Nov 03 13:30:06.364549 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:06.364498 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" podStartSLOduration=1.531637479 podStartE2EDuration="6.364484673s" totalImagesPullingTime="4.832847194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:06.363208734 +0000 UTC m=+1537.677180595" watchObservedRunningTime="2026-11-03 13:30:06.364484673 +0000 UTC m=+1537.678456533" Nov 03 13:30:07.135275 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:07.135238 4541 generic.go:356] "Generic (PLEG): container finished" podID="026b4e23-e769-4652-b452-2a884975d36e" containerID="5aa8476ab04f3e5bbd4d8f2184840467aaac442918b493cbd8a15b7f7233dbee" exitCode=60 Nov 03 13:30:07.135758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:07.135318 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" event={"ID":"026b4e23-e769-4652-b452-2a884975d36e","Type":"ContainerDied","Data":"5aa8476ab04f3e5bbd4d8f2184840467aaac442918b493cbd8a15b7f7233dbee"} Nov 03 13:30:07.135758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:07.135687 4541 scope.go:122] "RemoveContainer" containerID="5aa8476ab04f3e5bbd4d8f2184840467aaac442918b493cbd8a15b7f7233dbee" Nov 03 13:30:07.137631 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:07.137603 4541 generic.go:356] "Generic (PLEG): container finished" podID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerID="79fbed8e27dd1fcba09adde4aa7f4e5d36dbf32668a464e738c3d77dda8a137f" exitCode=60 Nov 03 13:30:07.137737 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:07.137652 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" event={"ID":"fcf391c9-d26c-408f-af2f-19a2cf151dec","Type":"ContainerDied","Data":"79fbed8e27dd1fcba09adde4aa7f4e5d36dbf32668a464e738c3d77dda8a137f"} Nov 03 13:30:07.137737 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:07.137713 4541 scope.go:122] "RemoveContainer" containerID="25ea87e1884e7e6d552847dbf59e2673c88c91ae01623b790679d1d0631131da" Nov 03 13:30:07.138026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:07.138008 4541 scope.go:122] "RemoveContainer" containerID="79fbed8e27dd1fcba09adde4aa7f4e5d36dbf32668a464e738c3d77dda8a137f" Nov 03 13:30:07.138355 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:30:07.138331 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7_odh-ai-gateway-infra(fcf391c9-d26c-408f-af2f-19a2cf151dec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" Nov 03 13:30:08.148157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:08.148117 4541 generic.go:356] "Generic (PLEG): container finished" podID="026b4e23-e769-4652-b452-2a884975d36e" containerID="630f20058d9b76bce23d70b8e3a4ce97d29f5666a4ca8b1135b2842814d44ae9" exitCode=60 Nov 03 13:30:08.148618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:08.148204 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" event={"ID":"026b4e23-e769-4652-b452-2a884975d36e","Type":"ContainerDied","Data":"630f20058d9b76bce23d70b8e3a4ce97d29f5666a4ca8b1135b2842814d44ae9"} Nov 03 13:30:08.148618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:08.148284 4541 scope.go:122] "RemoveContainer" containerID="5aa8476ab04f3e5bbd4d8f2184840467aaac442918b493cbd8a15b7f7233dbee" Nov 03 13:30:08.148618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:08.148581 4541 scope.go:122] "RemoveContainer" containerID="630f20058d9b76bce23d70b8e3a4ce97d29f5666a4ca8b1135b2842814d44ae9" Nov 03 13:30:08.149100 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:30:08.149069 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd_odh-ai-gateway-infra(026b4e23-e769-4652-b452-2a884975d36e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" podUID="026b4e23-e769-4652-b452-2a884975d36e" Nov 03 13:30:08.150678 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:08.150656 4541 scope.go:122] "RemoveContainer" containerID="79fbed8e27dd1fcba09adde4aa7f4e5d36dbf32668a464e738c3d77dda8a137f" Nov 03 13:30:08.151132 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:30:08.151073 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7_odh-ai-gateway-infra(fcf391c9-d26c-408f-af2f-19a2cf151dec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" Nov 03 13:30:09.160460 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:09.160433 4541 scope.go:122] "RemoveContainer" containerID="630f20058d9b76bce23d70b8e3a4ce97d29f5666a4ca8b1135b2842814d44ae9" Nov 03 13:30:09.160951 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:30:09.160741 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd_odh-ai-gateway-infra(026b4e23-e769-4652-b452-2a884975d36e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" podUID="026b4e23-e769-4652-b452-2a884975d36e" Nov 03 13:30:16.890607 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:30:16.890573 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:30:17.861532 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:17.861504 4541 scope.go:122] "RemoveContainer" containerID="79fbed8e27dd1fcba09adde4aa7f4e5d36dbf32668a464e738c3d77dda8a137f" Nov 03 13:30:18.225298 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:18.225255 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" event={"ID":"fcf391c9-d26c-408f-af2f-19a2cf151dec","Type":"ContainerStarted","Data":"d99458f0813d620a384843517d90e35f06b4a1e3ec2ba9a3f8b5e58c6c50b6db"} Nov 03 13:30:18.335767 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:18.335711 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" podStartSLOduration=14.100192747 podStartE2EDuration="18.335695868s" totalImagesPullingTime="4.235503121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:18.333381837 +0000 UTC m=+1549.647353699" watchObservedRunningTime="2026-11-03 13:30:18.335695868 +0000 UTC m=+1549.649668068" Nov 03 13:30:18.864799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:18.864740 4541 scope.go:122] "RemoveContainer" containerID="630f20058d9b76bce23d70b8e3a4ce97d29f5666a4ca8b1135b2842814d44ae9" Nov 03 13:30:19.235354 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:19.235259 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" event={"ID":"026b4e23-e769-4652-b452-2a884975d36e","Type":"ContainerStarted","Data":"3edfcd591581c21bacd7a7bb84826f8d065a6ec0cc8c08c98632c8f890b48a52"} Nov 03 13:30:19.237520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:19.237492 4541 generic.go:356] "Generic (PLEG): container finished" podID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerID="d99458f0813d620a384843517d90e35f06b4a1e3ec2ba9a3f8b5e58c6c50b6db" exitCode=60 Nov 03 13:30:19.237663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:19.237565 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" event={"ID":"fcf391c9-d26c-408f-af2f-19a2cf151dec","Type":"ContainerDied","Data":"d99458f0813d620a384843517d90e35f06b4a1e3ec2ba9a3f8b5e58c6c50b6db"} Nov 03 13:30:19.237663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:19.237633 4541 scope.go:122] "RemoveContainer" containerID="79fbed8e27dd1fcba09adde4aa7f4e5d36dbf32668a464e738c3d77dda8a137f" Nov 03 13:30:19.238068 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:19.238053 4541 scope.go:122] "RemoveContainer" containerID="d99458f0813d620a384843517d90e35f06b4a1e3ec2ba9a3f8b5e58c6c50b6db" Nov 03 13:30:19.238391 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:30:19.238370 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7_odh-ai-gateway-infra(fcf391c9-d26c-408f-af2f-19a2cf151dec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" Nov 03 13:30:19.345522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:19.345487 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7"] Nov 03 13:30:20.248991 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.248959 4541 generic.go:356] "Generic (PLEG): container finished" podID="026b4e23-e769-4652-b452-2a884975d36e" containerID="3edfcd591581c21bacd7a7bb84826f8d065a6ec0cc8c08c98632c8f890b48a52" exitCode=60 Nov 03 13:30:20.249393 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.249018 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" event={"ID":"026b4e23-e769-4652-b452-2a884975d36e","Type":"ContainerDied","Data":"3edfcd591581c21bacd7a7bb84826f8d065a6ec0cc8c08c98632c8f890b48a52"} Nov 03 13:30:20.249393 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.249077 4541 scope.go:122] "RemoveContainer" containerID="630f20058d9b76bce23d70b8e3a4ce97d29f5666a4ca8b1135b2842814d44ae9" Nov 03 13:30:20.249502 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.249486 4541 scope.go:122] "RemoveContainer" containerID="3edfcd591581c21bacd7a7bb84826f8d065a6ec0cc8c08c98632c8f890b48a52" Nov 03 13:30:20.249904 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:30:20.249874 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd_odh-ai-gateway-infra(026b4e23-e769-4652-b452-2a884975d36e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" podUID="026b4e23-e769-4652-b452-2a884975d36e" Nov 03 13:30:20.361755 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.361718 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd"] Nov 03 13:30:20.409967 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.409941 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:20.476852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.476824 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcf391c9-d26c-408f-af2f-19a2cf151dec-kube-api-access-kzp5z\" (UniqueName: \"kubernetes.io/projected/fcf391c9-d26c-408f-af2f-19a2cf151dec-kube-api-access-kzp5z\") pod \"fcf391c9-d26c-408f-af2f-19a2cf151dec\" (UID: \"fcf391c9-d26c-408f-af2f-19a2cf151dec\") " Nov 03 13:30:20.477079 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.476933 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcf391c9-d26c-408f-af2f-19a2cf151dec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf391c9-d26c-408f-af2f-19a2cf151dec-maas-api-service-ca\") pod \"fcf391c9-d26c-408f-af2f-19a2cf151dec\" (UID: \"fcf391c9-d26c-408f-af2f-19a2cf151dec\") " Nov 03 13:30:20.477320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.477293 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf391c9-d26c-408f-af2f-19a2cf151dec-maas-api-service-ca" pod "fcf391c9-d26c-408f-af2f-19a2cf151dec" (UID: "fcf391c9-d26c-408f-af2f-19a2cf151dec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:20.477435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.477422 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf391c9-d26c-408f-af2f-19a2cf151dec-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:20.479139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.479108 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcf391c9-d26c-408f-af2f-19a2cf151dec-kube-api-access-kzp5z" pod "fcf391c9-d26c-408f-af2f-19a2cf151dec" (UID: "fcf391c9-d26c-408f-af2f-19a2cf151dec"). InnerVolumeSpecName "kube-api-access-kzp5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:20.579231 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:20.579152 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzp5z\" (UniqueName: \"kubernetes.io/projected/fcf391c9-d26c-408f-af2f-19a2cf151dec-kube-api-access-kzp5z\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:21.257459 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.257429 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" Nov 03 13:30:21.257960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.257457 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7" event={"ID":"fcf391c9-d26c-408f-af2f-19a2cf151dec","Type":"ContainerDied","Data":"a29c9b7ff7bb1ca16d725dd71cfef097e7eb854d50bde77f15d7696037e73f21"} Nov 03 13:30:21.257960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.257525 4541 scope.go:122] "RemoveContainer" containerID="d99458f0813d620a384843517d90e35f06b4a1e3ec2ba9a3f8b5e58c6c50b6db" Nov 03 13:30:21.438579 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.438546 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:21.491127 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.491087 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/026b4e23-e769-4652-b452-2a884975d36e-kube-api-access-w9ztc\" (UniqueName: \"kubernetes.io/projected/026b4e23-e769-4652-b452-2a884975d36e-kube-api-access-w9ztc\") pod \"026b4e23-e769-4652-b452-2a884975d36e\" (UID: \"026b4e23-e769-4652-b452-2a884975d36e\") " Nov 03 13:30:21.491289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.491140 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/026b4e23-e769-4652-b452-2a884975d36e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/026b4e23-e769-4652-b452-2a884975d36e-maas-api-service-ca\") pod \"026b4e23-e769-4652-b452-2a884975d36e\" (UID: \"026b4e23-e769-4652-b452-2a884975d36e\") " Nov 03 13:30:21.491596 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.491567 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/026b4e23-e769-4652-b452-2a884975d36e-maas-api-service-ca" pod "026b4e23-e769-4652-b452-2a884975d36e" (UID: "026b4e23-e769-4652-b452-2a884975d36e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:21.491830 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.491789 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/026b4e23-e769-4652-b452-2a884975d36e-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:21.493330 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.493305 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/026b4e23-e769-4652-b452-2a884975d36e-kube-api-access-w9ztc" pod "026b4e23-e769-4652-b452-2a884975d36e" (UID: "026b4e23-e769-4652-b452-2a884975d36e"). InnerVolumeSpecName "kube-api-access-w9ztc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:21.523076 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.522998 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7"] Nov 03 13:30:21.535287 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.535255 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895210-kjkj7"] Nov 03 13:30:21.593240 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:21.593205 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9ztc\" (UniqueName: \"kubernetes.io/projected/026b4e23-e769-4652-b452-2a884975d36e-kube-api-access-w9ztc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:22.266348 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:22.266318 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" Nov 03 13:30:22.266838 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:22.266356 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd" event={"ID":"026b4e23-e769-4652-b452-2a884975d36e","Type":"ContainerDied","Data":"a7ebd76bce4fd76d3a4ab596a12b75e4a3b6ca6459b6a39fdc822513fb8544d3"} Nov 03 13:30:22.266838 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:22.266421 4541 scope.go:122] "RemoveContainer" containerID="3edfcd591581c21bacd7a7bb84826f8d065a6ec0cc8c08c98632c8f890b48a52" Nov 03 13:30:22.424780 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:22.424693 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd"] Nov 03 13:30:22.434063 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:22.434033 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-30074e-29895210-68smd"] Nov 03 13:30:22.869020 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:22.868990 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="026b4e23-e769-4652-b452-2a884975d36e" path="/var/lib/kubelet/pods/026b4e23-e769-4652-b452-2a884975d36e/volumes" Nov 03 13:30:22.869452 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:22.869438 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" path="/var/lib/kubelet/pods/fcf391c9-d26c-408f-af2f-19a2cf151dec/volumes" Nov 03 13:30:46.800603 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.800570 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t"] Nov 03 13:30:46.801328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801298 4541 cpu_manager.go:414] "containerMap: removing container" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerName="cleanup" Nov 03 13:30:46.801328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801327 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerName="cleanup" Nov 03 13:30:46.801497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801354 4541 cpu_manager.go:414] "containerMap: removing container" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerName="cleanup" Nov 03 13:30:46.801497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801382 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerName="cleanup" Nov 03 13:30:46.801497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801399 4541 cpu_manager.go:414] "containerMap: removing container" podUID="026b4e23-e769-4652-b452-2a884975d36e" containerName="cleanup" Nov 03 13:30:46.801497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801407 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026b4e23-e769-4652-b452-2a884975d36e" containerName="cleanup" Nov 03 13:30:46.801497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801414 4541 cpu_manager.go:414] "containerMap: removing container" podUID="026b4e23-e769-4652-b452-2a884975d36e" containerName="cleanup" Nov 03 13:30:46.801497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801420 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026b4e23-e769-4652-b452-2a884975d36e" containerName="cleanup" Nov 03 13:30:46.801497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801430 4541 cpu_manager.go:414] "containerMap: removing container" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerName="cleanup" Nov 03 13:30:46.801497 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801435 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerName="cleanup" Nov 03 13:30:46.801809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801575 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="026b4e23-e769-4652-b452-2a884975d36e" containerName="cleanup" Nov 03 13:30:46.801809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801590 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerName="cleanup" Nov 03 13:30:46.801809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801599 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerName="cleanup" Nov 03 13:30:46.801809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801608 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="026b4e23-e769-4652-b452-2a884975d36e" containerName="cleanup" Nov 03 13:30:46.801809 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.801617 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcf391c9-d26c-408f-af2f-19a2cf151dec" containerName="cleanup" Nov 03 13:30:46.810275 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.810257 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:46.872250 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.872214 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t"] Nov 03 13:30:46.894114 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:30:46.894073 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:30:46.903690 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.903664 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-28d87c-serving-cert\"" Nov 03 13:30:46.978348 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.978322 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-maas-api-tls\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:46.978527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.978471 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-metrics-certs\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:46.978527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.978508 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347b5f6e-0def-408a-9739-fceb7e2d647b-openshift-service-ca\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:46.978676 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:46.978659 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnhg\" (UniqueName: \"kubernetes.io/projected/347b5f6e-0def-408a-9739-fceb7e2d647b-kube-api-access-ghnhg\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.080278 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.080179 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnhg\" (UniqueName: \"kubernetes.io/projected/347b5f6e-0def-408a-9739-fceb7e2d647b-kube-api-access-ghnhg\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.080278 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.080258 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-maas-api-tls\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.080535 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.080347 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-metrics-certs\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.080535 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.080366 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347b5f6e-0def-408a-9739-fceb7e2d647b-openshift-service-ca\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.081090 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.081065 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347b5f6e-0def-408a-9739-fceb7e2d647b-openshift-service-ca\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.082759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.082736 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-maas-api-tls\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.082867 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.082843 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-metrics-certs\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.135179 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.135148 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnhg\" (UniqueName: \"kubernetes.io/projected/347b5f6e-0def-408a-9739-fceb7e2d647b-kube-api-access-ghnhg\") pod \"maas-api-e2e-api-b-28d87c-bf645f655-xfw8t\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.423933 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.423902 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:47.796176 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:47.796149 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t"] Nov 03 13:30:47.796975 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:30:47.796947 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347b5f6e_0def_408a_9739_fceb7e2d647b.slice/crio-f539f2539ace2acfe040383790b5d49d01ba1e785dec64cd604057f910e07e92 WatchSource:0}: Error finding container f539f2539ace2acfe040383790b5d49d01ba1e785dec64cd604057f910e07e92: Status 404 returned error can't find the container with id f539f2539ace2acfe040383790b5d49d01ba1e785dec64cd604057f910e07e92 Nov 03 13:30:48.430999 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:48.430956 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" event={"ID":"347b5f6e-0def-408a-9739-fceb7e2d647b","Type":"ContainerStarted","Data":"f539f2539ace2acfe040383790b5d49d01ba1e785dec64cd604057f910e07e92"} Nov 03 13:30:49.440440 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:49.440390 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" event={"ID":"347b5f6e-0def-408a-9739-fceb7e2d647b","Type":"ContainerStarted","Data":"2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65"} Nov 03 13:30:49.440937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:49.440468 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:30:49.552864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:49.552807 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" podStartSLOduration=2.648901792 podStartE2EDuration="3.552793304s" totalImagesPullingTime="903.891512ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:49.549523099 +0000 UTC m=+1580.863494959" watchObservedRunningTime="2026-11-03 13:30:49.552793304 +0000 UTC m=+1580.866765164" Nov 03 13:30:55.453932 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:30:55.453901 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:31:07.563466 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:07.563426 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r"] Nov 03 13:31:07.564021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:07.563688 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" containerID="cri-o://ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23" gracePeriod=30 Nov 03 13:31:07.610232 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:07.610196 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz"] Nov 03 13:31:07.610445 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:07.610408 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" containerID="cri-o://a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03" gracePeriod=30 Nov 03 13:31:12.876705 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:12.876668 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:31:12.939928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:12.939902 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:31:13.034988 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.034956 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-podinfo\") pod \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " Nov 03 13:31:13.034988 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.034992 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-istio-token\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-istio-token\") pod \"9046e25c-55b4-4746-a618-8e8b48a69641\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " Nov 03 13:31:13.035213 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035028 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-data\") pod \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " Nov 03 13:31:13.035213 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035068 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-credential-socket\") pod \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " Nov 03 13:31:13.035213 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035111 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9046e25c-55b4-4746-a618-8e8b48a69641-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9046e25c-55b4-4746-a618-8e8b48a69641-istio-podinfo\") pod \"9046e25c-55b4-4746-a618-8e8b48a69641\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " Nov 03 13:31:13.035213 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035146 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-credential-socket\") pod \"9046e25c-55b4-4746-a618-8e8b48a69641\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " Nov 03 13:31:13.035213 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035186 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-certs\") pod \"9046e25c-55b4-4746-a618-8e8b48a69641\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " Nov 03 13:31:13.035476 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035227 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-envoy\") pod \"9046e25c-55b4-4746-a618-8e8b48a69641\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " Nov 03 13:31:13.035476 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035343 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-data\") pod \"9046e25c-55b4-4746-a618-8e8b48a69641\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " Nov 03 13:31:13.035476 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035374 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-credential-socket" pod "3c4e92b6-c318-43f8-8a8e-e7c421895a49" (UID: "3c4e92b6-c318-43f8-8a8e-e7c421895a49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.035476 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035383 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-data" pod "3c4e92b6-c318-43f8-8a8e-e7c421895a49" (UID: "3c4e92b6-c318-43f8-8a8e-e7c421895a49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.035476 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035390 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-kube-api-access-r625k\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-kube-api-access-r625k\") pod \"9046e25c-55b4-4746-a618-8e8b48a69641\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " Nov 03 13:31:13.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035498 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-certs" pod "9046e25c-55b4-4746-a618-8e8b48a69641" (UID: "9046e25c-55b4-4746-a618-8e8b48a69641"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035510 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-credential-socket" pod "9046e25c-55b4-4746-a618-8e8b48a69641" (UID: "9046e25c-55b4-4746-a618-8e8b48a69641"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035537 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-kube-api-access-pwlwn\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-kube-api-access-pwlwn\") pod \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " Nov 03 13:31:13.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035589 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-envoy\") pod \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " Nov 03 13:31:13.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035629 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istiod-ca-cert\") pod \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " Nov 03 13:31:13.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035662 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-token\") pod \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " Nov 03 13:31:13.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035690 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9046e25c-55b4-4746-a618-8e8b48a69641-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9046e25c-55b4-4746-a618-8e8b48a69641-istiod-ca-cert\") pod \"9046e25c-55b4-4746-a618-8e8b48a69641\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " Nov 03 13:31:13.035734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035710 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-data" pod "9046e25c-55b4-4746-a618-8e8b48a69641" (UID: "9046e25c-55b4-4746-a618-8e8b48a69641"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.036190 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035808 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-socket\") pod \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " Nov 03 13:31:13.036190 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035848 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-certs\") pod \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\" (UID: \"3c4e92b6-c318-43f8-8a8e-e7c421895a49\") " Nov 03 13:31:13.036190 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.035906 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-socket\") pod \"9046e25c-55b4-4746-a618-8e8b48a69641\" (UID: \"9046e25c-55b4-4746-a618-8e8b48a69641\") " Nov 03 13:31:13.036190 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036057 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istiod-ca-cert" pod "3c4e92b6-c318-43f8-8a8e-e7c421895a49" (UID: "3c4e92b6-c318-43f8-8a8e-e7c421895a49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:13.036427 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036319 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9046e25c-55b4-4746-a618-8e8b48a69641-istiod-ca-cert" pod "9046e25c-55b4-4746-a618-8e8b48a69641" (UID: "9046e25c-55b4-4746-a618-8e8b48a69641"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:13.036716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036551 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-socket" pod "3c4e92b6-c318-43f8-8a8e-e7c421895a49" (UID: "3c4e92b6-c318-43f8-8a8e-e7c421895a49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.036716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036688 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-certs" pod "3c4e92b6-c318-43f8-8a8e-e7c421895a49" (UID: "3c4e92b6-c318-43f8-8a8e-e7c421895a49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.037144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036784 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.037144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036809 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.037144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036824 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.037144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036840 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.037144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036856 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9046e25c-55b4-4746-a618-8e8b48a69641-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.037144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036870 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.037144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036885 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.037144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036899 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.037144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.036966 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-socket" pod "9046e25c-55b4-4746-a618-8e8b48a69641" (UID: "9046e25c-55b4-4746-a618-8e8b48a69641"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.038138 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.038108 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9046e25c-55b4-4746-a618-8e8b48a69641-istio-podinfo" pod "9046e25c-55b4-4746-a618-8e8b48a69641" (UID: "9046e25c-55b4-4746-a618-8e8b48a69641"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:13.038282 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.038246 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-envoy" pod "9046e25c-55b4-4746-a618-8e8b48a69641" (UID: "9046e25c-55b4-4746-a618-8e8b48a69641"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.038735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.038708 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-kube-api-access-r625k" pod "9046e25c-55b4-4746-a618-8e8b48a69641" (UID: "9046e25c-55b4-4746-a618-8e8b48a69641"). InnerVolumeSpecName "kube-api-access-r625k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:13.038855 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.038719 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-kube-api-access-pwlwn" pod "3c4e92b6-c318-43f8-8a8e-e7c421895a49" (UID: "3c4e92b6-c318-43f8-8a8e-e7c421895a49"). InnerVolumeSpecName "kube-api-access-pwlwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:13.039377 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.039344 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-envoy" pod "3c4e92b6-c318-43f8-8a8e-e7c421895a49" (UID: "3c4e92b6-c318-43f8-8a8e-e7c421895a49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.039567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.039544 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-istio-token" pod "9046e25c-55b4-4746-a618-8e8b48a69641" (UID: "9046e25c-55b4-4746-a618-8e8b48a69641"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:13.039759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.039736 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-podinfo" pod "3c4e92b6-c318-43f8-8a8e-e7c421895a49" (UID: "3c4e92b6-c318-43f8-8a8e-e7c421895a49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:13.040056 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.040021 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-token" pod "3c4e92b6-c318-43f8-8a8e-e7c421895a49" (UID: "3c4e92b6-c318-43f8-8a8e-e7c421895a49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:13.138117 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138021 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwlwn\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-kube-api-access-pwlwn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.138117 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138054 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.138117 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138068 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.138117 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138080 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4e92b6-c318-43f8-8a8e-e7c421895a49-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.138117 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138092 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.138117 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138104 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4e92b6-c318-43f8-8a8e-e7c421895a49-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.138117 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138118 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.138457 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138130 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9046e25c-55b4-4746-a618-8e8b48a69641-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.138457 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138142 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9046e25c-55b4-4746-a618-8e8b48a69641-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.138457 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.138154 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r625k\" (UniqueName: \"kubernetes.io/projected/9046e25c-55b4-4746-a618-8e8b48a69641-kube-api-access-r625k\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.605209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.605174 4541 generic.go:356] "Generic (PLEG): container finished" podID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerID="ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23" exitCode=0 Nov 03 13:31:13.605354 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.605219 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" Nov 03 13:31:13.605354 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.605257 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" event={"ID":"3c4e92b6-c318-43f8-8a8e-e7c421895a49","Type":"ContainerDied","Data":"ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23"} Nov 03 13:31:13.605354 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.605315 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r" event={"ID":"3c4e92b6-c318-43f8-8a8e-e7c421895a49","Type":"ContainerDied","Data":"85e1c9f27d1409608e38b78dcd5c01f6e3909db7fe877de994d6005d549c2ed8"} Nov 03 13:31:13.605354 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.605339 4541 scope.go:122] "RemoveContainer" containerID="ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23" Nov 03 13:31:13.606672 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.606648 4541 generic.go:356] "Generic (PLEG): container finished" podID="9046e25c-55b4-4746-a618-8e8b48a69641" containerID="a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03" exitCode=0 Nov 03 13:31:13.606822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.606699 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" Nov 03 13:31:13.606822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.606741 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" event={"ID":"9046e25c-55b4-4746-a618-8e8b48a69641","Type":"ContainerDied","Data":"a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03"} Nov 03 13:31:13.606822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.606806 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz" event={"ID":"9046e25c-55b4-4746-a618-8e8b48a69641","Type":"ContainerDied","Data":"6dce0a26aed16f485038fa784f075badce6aa36a1f8ce4a5f489b13ee0c5a4e4"} Nov 03 13:31:13.615740 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.615723 4541 scope.go:122] "RemoveContainer" containerID="ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23" Nov 03 13:31:13.616101 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:31:13.616082 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23\": container with ID starting with ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23 not found: ID does not exist" containerID="ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23" Nov 03 13:31:13.616193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.616124 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23"} err="failed to get container status \"ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23\": rpc error: code = NotFound desc = could not find container \"ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23\": container with ID starting with ca99f5b7130fb4529704a41f1e1fac25475b60fac046b34dfdf38512dca59d23 not found: ID does not exist" Nov 03 13:31:13.616193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.616151 4541 scope.go:122] "RemoveContainer" containerID="a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03" Nov 03 13:31:13.625289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.625273 4541 scope.go:122] "RemoveContainer" containerID="a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03" Nov 03 13:31:13.625555 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:31:13.625536 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03\": container with ID starting with a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03 not found: ID does not exist" containerID="a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03" Nov 03 13:31:13.625630 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.625561 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03"} err="failed to get container status \"a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03\": rpc error: code = NotFound desc = could not find container \"a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03\": container with ID starting with a292733a359bf38a6707448038425e427f33b02b6628b7b8f3b4f403f819db03 not found: ID does not exist" Nov 03 13:31:13.762109 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.762064 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz"] Nov 03 13:31:13.809194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.809165 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-zq9nz"] Nov 03 13:31:13.871305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.871125 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r"] Nov 03 13:31:13.874966 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:13.874938 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-rmj7r"] Nov 03 13:31:14.869170 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:14.869138 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" path="/var/lib/kubelet/pods/3c4e92b6-c318-43f8-8a8e-e7c421895a49/volumes" Nov 03 13:31:14.869596 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:14.869582 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" path="/var/lib/kubelet/pods/9046e25c-55b4-4746-a618-8e8b48a69641/volumes" Nov 03 13:31:16.890470 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:31:16.890434 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:31:32.448945 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.448911 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r"] Nov 03 13:31:32.449507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.449491 4541 cpu_manager.go:414] "containerMap: removing container" podUID="026b4e23-e769-4652-b452-2a884975d36e" containerName="cleanup" Nov 03 13:31:32.449550 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.449508 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="026b4e23-e769-4652-b452-2a884975d36e" containerName="cleanup" Nov 03 13:31:32.449550 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.449533 4541 cpu_manager.go:414] "containerMap: removing container" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" Nov 03 13:31:32.449550 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.449543 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" Nov 03 13:31:32.449641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.449557 4541 cpu_manager.go:414] "containerMap: removing container" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" Nov 03 13:31:32.449641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.449563 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" Nov 03 13:31:32.449705 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.449676 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4e92b6-c318-43f8-8a8e-e7c421895a49" containerName="istio-proxy" Nov 03 13:31:32.449705 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.449689 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="9046e25c-55b4-4746-a618-8e8b48a69641" containerName="istio-proxy" Nov 03 13:31:32.449705 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.449701 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="026b4e23-e769-4652-b452-2a884975d36e" containerName="cleanup" Nov 03 13:31:32.454365 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.454346 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.503958 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.503928 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-28d87c-openshift-default-dockercfg-lcrzm\"" Nov 03 13:31:32.618096 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.618055 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.618293 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.618112 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.618365 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.618294 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7s8j\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-kube-api-access-p7s8j\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.618405 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.618395 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112a1add-c290-4f08-b1c5-48118ce06a2b-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.618459 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.618426 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.618522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.618500 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.618591 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.618575 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.618633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.618608 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.618672 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.618656 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.662021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.661988 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r"] Nov 03 13:31:32.719891 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.719812 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7s8j\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-kube-api-access-p7s8j\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.719891 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.719884 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112a1add-c290-4f08-b1c5-48118ce06a2b-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.719918 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.719951 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720071 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720108 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720163 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720277 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720439 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720335 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720439 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720384 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720624 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720602 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112a1add-c290-4f08-b1c5-48118ce06a2b-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720680 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720663 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720737 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720714 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.720886 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.720867 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.722593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.722574 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.722593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.722585 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.774638 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.774604 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:32.774884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:32.774858 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7s8j\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-kube-api-access-p7s8j\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-zj29r\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:33.067728 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:33.067638 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:33.489566 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:33.489532 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r"] Nov 03 13:31:33.489956 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:31:33.489634 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112a1add_c290_4f08_b1c5_48118ce06a2b.slice/crio-9ce0dc0c9614472bb746bb4af267ccaec1f24fbe23cc3d88e3f8cd2738c346d0 WatchSource:0}: Error finding container 9ce0dc0c9614472bb746bb4af267ccaec1f24fbe23cc3d88e3f8cd2738c346d0: Status 404 returned error can't find the container with id 9ce0dc0c9614472bb746bb4af267ccaec1f24fbe23cc3d88e3f8cd2738c346d0 Nov 03 13:31:33.492214 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:33.492182 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:31:33.492311 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:33.492250 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:31:33.492311 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:33.492294 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:31:33.740501 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:33.740409 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" event={"ID":"112a1add-c290-4f08-b1c5-48118ce06a2b","Type":"ContainerStarted","Data":"b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8"} Nov 03 13:31:33.740501 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:33.740476 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" event={"ID":"112a1add-c290-4f08-b1c5-48118ce06a2b","Type":"ContainerStarted","Data":"9ce0dc0c9614472bb746bb4af267ccaec1f24fbe23cc3d88e3f8cd2738c346d0"} Nov 03 13:31:33.853492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:33.853435 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" podStartSLOduration=1.853421859 podStartE2EDuration="1.853421859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:33.850385932 +0000 UTC m=+1625.164357805" watchObservedRunningTime="2026-11-03 13:31:33.853421859 +0000 UTC m=+1625.167393719" Nov 03 13:31:34.068554 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:34.068470 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:34.070107 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:34.070076 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:31:34.070177 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:34.070136 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 13:31:35.068655 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:35.068612 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:31:35.069117 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:35.068689 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 13:31:36.068316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:36.068277 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:31:36.068503 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:36.068347 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 13:31:37.068717 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:37.068681 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:31:37.069239 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:37.068742 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 13:31:38.068319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:38.068279 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:31:38.068508 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:38.068337 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 13:31:38.775810 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:38.768451 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt"] Nov 03 13:31:38.775810 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:38.768715 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" containerID="cri-o://f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83" gracePeriod=30 Nov 03 13:31:39.068580 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:39.068486 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 13:31:39.068580 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:39.068563 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 13:31:40.072650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:40.072616 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:40.073031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:40.072988 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:40.073733 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:40.073712 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:31:43.199755 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:43.199716 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:31:43.200179 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:43.199810 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:31:44.061413 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.061352 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:31:44.216070 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216022 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-podinfo\") pod \"2ce84d7a-3f43-41b5-947e-57d90b701707\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " Nov 03 13:31:44.216528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216122 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-certs\") pod \"2ce84d7a-3f43-41b5-947e-57d90b701707\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " Nov 03 13:31:44.216528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216189 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-envoy\") pod \"2ce84d7a-3f43-41b5-947e-57d90b701707\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " Nov 03 13:31:44.216528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216217 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-socket\") pod \"2ce84d7a-3f43-41b5-947e-57d90b701707\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " Nov 03 13:31:44.216528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216294 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-data\") pod \"2ce84d7a-3f43-41b5-947e-57d90b701707\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " Nov 03 13:31:44.216528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216378 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-kube-api-access-xrhzn\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-kube-api-access-xrhzn\") pod \"2ce84d7a-3f43-41b5-947e-57d90b701707\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " Nov 03 13:31:44.216528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216458 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-token\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-token\") pod \"2ce84d7a-3f43-41b5-947e-57d90b701707\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " Nov 03 13:31:44.216528 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216488 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ce84d7a-3f43-41b5-947e-57d90b701707-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce84d7a-3f43-41b5-947e-57d90b701707-istiod-ca-cert\") pod \"2ce84d7a-3f43-41b5-947e-57d90b701707\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " Nov 03 13:31:44.216981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216555 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-credential-socket\") pod \"2ce84d7a-3f43-41b5-947e-57d90b701707\" (UID: \"2ce84d7a-3f43-41b5-947e-57d90b701707\") " Nov 03 13:31:44.216981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216699 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-socket" pod "2ce84d7a-3f43-41b5-947e-57d90b701707" (UID: "2ce84d7a-3f43-41b5-947e-57d90b701707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.216981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.216904 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-certs" pod "2ce84d7a-3f43-41b5-947e-57d90b701707" (UID: "2ce84d7a-3f43-41b5-947e-57d90b701707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.217145 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.217090 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-data" pod "2ce84d7a-3f43-41b5-947e-57d90b701707" (UID: "2ce84d7a-3f43-41b5-947e-57d90b701707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.217200 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.217166 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.217200 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.217190 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.217631 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.217604 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-credential-socket" pod "2ce84d7a-3f43-41b5-947e-57d90b701707" (UID: "2ce84d7a-3f43-41b5-947e-57d90b701707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.217960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.217921 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce84d7a-3f43-41b5-947e-57d90b701707-istiod-ca-cert" pod "2ce84d7a-3f43-41b5-947e-57d90b701707" (UID: "2ce84d7a-3f43-41b5-947e-57d90b701707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:44.219323 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.219295 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-podinfo" pod "2ce84d7a-3f43-41b5-947e-57d90b701707" (UID: "2ce84d7a-3f43-41b5-947e-57d90b701707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:44.219633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.219614 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-kube-api-access-xrhzn" pod "2ce84d7a-3f43-41b5-947e-57d90b701707" (UID: "2ce84d7a-3f43-41b5-947e-57d90b701707"). InnerVolumeSpecName "kube-api-access-xrhzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.220086 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.220067 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-token" pod "2ce84d7a-3f43-41b5-947e-57d90b701707" (UID: "2ce84d7a-3f43-41b5-947e-57d90b701707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.220591 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.220576 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-envoy" pod "2ce84d7a-3f43-41b5-947e-57d90b701707" (UID: "2ce84d7a-3f43-41b5-947e-57d90b701707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.318861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.318751 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.318861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.318800 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrhzn\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-kube-api-access-xrhzn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.318861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.318816 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.318861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.318828 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ce84d7a-3f43-41b5-947e-57d90b701707-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.318861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.318841 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.318861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.318853 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.319148 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.318866 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ce84d7a-3f43-41b5-947e-57d90b701707-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.833273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.833238 4541 generic.go:356] "Generic (PLEG): container finished" podID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerID="f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83" exitCode=0 Nov 03 13:31:44.833475 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.833299 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" Nov 03 13:31:44.833475 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.833319 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" event={"ID":"2ce84d7a-3f43-41b5-947e-57d90b701707","Type":"ContainerDied","Data":"f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83"} Nov 03 13:31:44.833475 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.833413 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt" event={"ID":"2ce84d7a-3f43-41b5-947e-57d90b701707","Type":"ContainerDied","Data":"da63e20d2afea1cc9d996b7088c4e9a2ef884280d1498ab0c68e359b4bbc0d75"} Nov 03 13:31:44.833475 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.833446 4541 scope.go:122] "RemoveContainer" containerID="f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83" Nov 03 13:31:44.844200 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.844187 4541 scope.go:122] "RemoveContainer" containerID="f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83" Nov 03 13:31:44.844473 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:31:44.844454 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83\": container with ID starting with f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83 not found: ID does not exist" containerID="f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83" Nov 03 13:31:44.844513 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:44.844483 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83"} err="failed to get container status \"f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83\": rpc error: code = NotFound desc = could not find container \"f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83\": container with ID starting with f2c813f136f42c2accdb12cf500a942bb3439cd074c3ff1d22678528ef5e9d83 not found: ID does not exist" Nov 03 13:31:45.039277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:45.039236 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt"] Nov 03 13:31:45.043630 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:45.043600 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-qwqrt"] Nov 03 13:31:46.869409 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:46.869374 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" path="/var/lib/kubelet/pods/2ce84d7a-3f43-41b5-947e-57d90b701707/volumes" Nov 03 13:31:46.891354 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:31:46.891318 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:31:53.524228 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:53.524185 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:31:53.526567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:53.526527 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:58.925792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:58.925745 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29"] Nov 03 13:31:58.926663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:58.926633 4541 cpu_manager.go:414] "containerMap: removing container" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" Nov 03 13:31:58.926663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:58.926662 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" Nov 03 13:31:58.926954 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:58.926861 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ce84d7a-3f43-41b5-947e-57d90b701707" containerName="istio-proxy" Nov 03 13:31:58.930632 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:58.930611 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.023686 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.023641 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.023895 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.023720 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.023895 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.023842 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5864d40d-7211-41bd-990d-45d0e909224b-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.024025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.023901 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.024085 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.024054 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.024144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.024107 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5864d40d-7211-41bd-990d-45d0e909224b-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.024201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.024149 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crm88\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-kube-api-access-crm88\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.024255 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.024207 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.024372 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.024348 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.033167 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.033135 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29"] Nov 03 13:31:59.125339 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125303 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.125546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125356 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.125546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125459 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5864d40d-7211-41bd-990d-45d0e909224b-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.125546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125533 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.125734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125618 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.125734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125650 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5864d40d-7211-41bd-990d-45d0e909224b-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.125734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125701 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crm88\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-kube-api-access-crm88\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.125921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125740 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.125921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125748 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.126036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.125914 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.126093 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.126021 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.126232 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.126205 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.126300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.126274 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.126451 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.126336 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5864d40d-7211-41bd-990d-45d0e909224b-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.127988 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.127963 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.128153 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.128135 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5864d40d-7211-41bd-990d-45d0e909224b-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.180030 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.179961 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.180169 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.180113 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crm88\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-kube-api-access-crm88\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.244514 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.244483 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:31:59.695833 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.682756 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29"] Nov 03 13:31:59.736716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.730403 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:31:59.736716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.730506 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:31:59.736716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.730556 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:31:59.980354 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.980245 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" event={"ID":"5864d40d-7211-41bd-990d-45d0e909224b","Type":"ContainerStarted","Data":"28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c"} Nov 03 13:31:59.980721 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:31:59.980354 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" event={"ID":"5864d40d-7211-41bd-990d-45d0e909224b","Type":"ContainerStarted","Data":"febe7f86c0bfc9bb522739c6c3423437b54ee5eff773ac8facb57b2d2e039ac9"} Nov 03 13:32:00.161372 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:00.161301 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" podStartSLOduration=2.161280966 podStartE2EDuration="2.161280966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:00.147989806 +0000 UTC m=+1651.461961668" watchObservedRunningTime="2026-11-03 13:32:00.161280966 +0000 UTC m=+1651.475252827" Nov 03 13:32:00.246333 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:00.246235 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:32:00.246485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:00.246398 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 13:32:00.246485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:00.246459 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 13:32:01.255794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:01.253219 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 13:32:01.255794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:01.253285 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 13:32:02.260798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.258179 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 13:32:02.260798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.258238 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 13:32:02.732126 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.732084 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv"] Nov 03 13:32:02.765796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.751267 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.795742 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.795711 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.795742 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.795745 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.796001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.795899 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.796001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.795939 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.796118 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.796058 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.796118 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.796105 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dj4q\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-kube-api-access-5dj4q\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.796250 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.796228 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017540c-faef-4bae-ab8e-ca161785ad52-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.796338 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.796325 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017540c-faef-4bae-ab8e-ca161785ad52-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.796380 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.796372 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.878337 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.878307 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv"] Nov 03 13:32:02.897921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.897893 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017540c-faef-4bae-ab8e-ca161785ad52-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.898089 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.898057 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.898148 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.898134 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.898191 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.898160 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.898314 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.898292 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.898525 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.898506 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.899677 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.899156 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.899677 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.899221 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.899677 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.899385 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.899677 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.899419 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.899677 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.899465 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dj4q\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-kube-api-access-5dj4q\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.899677 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.899545 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017540c-faef-4bae-ab8e-ca161785ad52-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.900107 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.899883 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.901271 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.901244 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017540c-faef-4bae-ab8e-ca161785ad52-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.902641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.902619 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.911181 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.911155 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017540c-faef-4bae-ab8e-ca161785ad52-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.960587 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.960551 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dj4q\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-kube-api-access-5dj4q\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:02.960904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:02.960881 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:03.091916 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:03.091822 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:03.263867 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:03.263820 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 13:32:03.264326 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:03.263886 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 13:32:03.570463 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:03.570426 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv"] Nov 03 13:32:03.574039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:03.574006 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:03.574124 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:03.574071 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:03.574124 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:03.574103 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:04.004973 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:04.004932 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" event={"ID":"e017540c-faef-4bae-ab8e-ca161785ad52","Type":"ContainerStarted","Data":"eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48"} Nov 03 13:32:04.005175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:04.005006 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" event={"ID":"e017540c-faef-4bae-ab8e-ca161785ad52","Type":"ContainerStarted","Data":"dd144e74e146ffffb15bdba2e96e90950b0b9fd04e4aad757f8c7c69ed3dcc41"} Nov 03 13:32:04.093153 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:04.092828 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:04.094131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:04.094107 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 13:32:04.094226 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:04.094165 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 13:32:04.131910 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:04.131843 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" podStartSLOduration=2.131824601 podStartE2EDuration="2.131824601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:04.129429638 +0000 UTC m=+1655.443401499" watchObservedRunningTime="2026-11-03 13:32:04.131824601 +0000 UTC m=+1655.445796461" Nov 03 13:32:04.256852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:04.246195 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 13:32:04.256852 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:04.246259 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 13:32:05.114793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:05.103220 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 13:32:05.114793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:05.103288 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 13:32:05.245792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:05.245234 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 13:32:05.245792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:05.245300 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 13:32:06.092449 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:06.092410 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 13:32:06.092750 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:06.092479 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 13:32:06.245203 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:06.245164 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 13:32:06.245754 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:06.245721 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 13:32:07.093151 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:07.093105 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 13:32:07.093366 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:07.093172 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 13:32:07.494289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:07.494252 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:32:07.495240 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:07.495212 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:32:07.496503 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:07.496479 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:32:08.092966 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:08.092930 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 13:32:08.093124 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:08.092984 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 13:32:09.093126 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:09.093089 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 13:32:09.093549 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:09.093153 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 13:32:10.092913 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:10.092826 4541 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 13:32:10.092913 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:10.092894 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 13:32:11.097412 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:11.097378 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:11.097885 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:11.097676 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:11.098438 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:11.098419 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:32:16.891254 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:32:16.891220 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:32:25.133997 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.133961 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8"] Nov 03 13:32:25.138525 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.138503 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.169644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.169609 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.169832 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.169665 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.169832 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.169680 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.169928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.169843 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.169970 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.169952 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.170011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.169978 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.170046 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.170021 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.170165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.170147 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7jwf\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-kube-api-access-n7jwf\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.170218 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.170197 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271401 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271353 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271562 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271411 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271562 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271490 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271643 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271591 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271643 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271615 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271767 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271655 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271869 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271843 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7jwf\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-kube-api-access-n7jwf\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271875 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271882 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.271930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271914 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.272102 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271974 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.272102 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.271976 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.272223 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.272199 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.272595 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.272567 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.274217 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.274192 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.274554 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.274529 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.298451 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.298426 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5295ab29-openshift-default-dockercfg-h5sdt\"" Nov 03 13:32:25.326448 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.326419 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.326593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.326469 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7jwf\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-kube-api-access-n7jwf\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.339860 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.339829 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8"] Nov 03 13:32:25.452939 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.452906 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:25.876534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.876502 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8"] Nov 03 13:32:25.877492 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:32:25.877460 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef1e1f4_063c_4daa_8a31_f40d7bd561d5.slice/crio-549e6b26be5076b28553f593e141809c5801aef21595bb16f268bbdd782f3a2f WatchSource:0}: Error finding container 549e6b26be5076b28553f593e141809c5801aef21595bb16f268bbdd782f3a2f: Status 404 returned error can't find the container with id 549e6b26be5076b28553f593e141809c5801aef21595bb16f268bbdd782f3a2f Nov 03 13:32:25.883550 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.883517 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:25.883632 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.883588 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:25.883632 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:25.883617 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:26.184214 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:26.184174 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" event={"ID":"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5","Type":"ContainerStarted","Data":"a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8"} Nov 03 13:32:26.184729 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:26.184241 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" event={"ID":"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5","Type":"ContainerStarted","Data":"549e6b26be5076b28553f593e141809c5801aef21595bb16f268bbdd782f3a2f"} Nov 03 13:32:26.299300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:26.299234 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" podStartSLOduration=1.2992162170000001 podStartE2EDuration="1.299216217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:26.298034514 +0000 UTC m=+1677.612006388" watchObservedRunningTime="2026-11-03 13:32:26.299216217 +0000 UTC m=+1677.613188078" Nov 03 13:32:26.457619 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:26.457515 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:26.459461 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:26.459424 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:32:26.459616 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:26.459486 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 13:32:27.456794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:27.456157 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:32:27.456794 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:27.456234 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 13:32:28.460316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:28.460256 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:32:28.460883 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:28.460329 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 13:32:29.131790 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:29.131746 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd"] Nov 03 13:32:29.131978 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:29.131956 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" containerID="cri-o://c051e2e60cdb4b5954a98215304609ed02d88caa348b209ad41cd7ebd013afde" gracePeriod=30 Nov 03 13:32:29.459796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:29.458698 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:32:29.459796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:29.458756 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 13:32:30.457796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:30.456871 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:32:30.457796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:30.456930 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 13:32:31.457793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:31.457337 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 13:32:31.457793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:31.457494 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 13:32:32.460971 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:32.460942 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:32.461381 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:32.461359 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:32.461997 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:32.461981 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:32:34.249157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.249058 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" event={"ID":"16cafd0c-7aa9-498d-8f7b-ca52431af32d","Type":"ContainerDied","Data":"c051e2e60cdb4b5954a98215304609ed02d88caa348b209ad41cd7ebd013afde"} Nov 03 13:32:34.249157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.249104 4541 generic.go:356] "Generic (PLEG): container finished" podID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerID="c051e2e60cdb4b5954a98215304609ed02d88caa348b209ad41cd7ebd013afde" exitCode=0 Nov 03 13:32:34.330250 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.330227 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:32:34.341393 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341368 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-socket\") pod \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " Nov 03 13:32:34.341525 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341408 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-kube-api-access-rr2pv\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-kube-api-access-rr2pv\") pod \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " Nov 03 13:32:34.341525 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341505 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-data\") pod \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " Nov 03 13:32:34.341618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341525 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-token\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-token\") pod \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " Nov 03 13:32:34.341618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341598 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istiod-ca-cert\") pod \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " Nov 03 13:32:34.341724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341642 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-podinfo\") pod \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " Nov 03 13:32:34.341724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341671 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-certs\") pod \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " Nov 03 13:32:34.341724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341673 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-socket" pod "16cafd0c-7aa9-498d-8f7b-ca52431af32d" (UID: "16cafd0c-7aa9-498d-8f7b-ca52431af32d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.341896 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341834 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-envoy\") pod \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " Nov 03 13:32:34.341896 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341883 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-credential-socket\") pod \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\" (UID: \"16cafd0c-7aa9-498d-8f7b-ca52431af32d\") " Nov 03 13:32:34.342007 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.341889 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-data" pod "16cafd0c-7aa9-498d-8f7b-ca52431af32d" (UID: "16cafd0c-7aa9-498d-8f7b-ca52431af32d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.342085 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.342056 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istiod-ca-cert" pod "16cafd0c-7aa9-498d-8f7b-ca52431af32d" (UID: "16cafd0c-7aa9-498d-8f7b-ca52431af32d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:34.342085 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.342058 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-certs" pod "16cafd0c-7aa9-498d-8f7b-ca52431af32d" (UID: "16cafd0c-7aa9-498d-8f7b-ca52431af32d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.342311 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.342290 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-credential-socket" pod "16cafd0c-7aa9-498d-8f7b-ca52431af32d" (UID: "16cafd0c-7aa9-498d-8f7b-ca52431af32d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.342724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.342703 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.342724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.342722 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.342921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.342732 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.342921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.342742 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.342921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.342752 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.344384 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.344358 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-podinfo" pod "16cafd0c-7aa9-498d-8f7b-ca52431af32d" (UID: "16cafd0c-7aa9-498d-8f7b-ca52431af32d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:32:34.344474 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.344405 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-envoy" pod "16cafd0c-7aa9-498d-8f7b-ca52431af32d" (UID: "16cafd0c-7aa9-498d-8f7b-ca52431af32d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.344474 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.344440 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-kube-api-access-rr2pv" pod "16cafd0c-7aa9-498d-8f7b-ca52431af32d" (UID: "16cafd0c-7aa9-498d-8f7b-ca52431af32d"). InnerVolumeSpecName "kube-api-access-rr2pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.344546 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.344504 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-token" pod "16cafd0c-7aa9-498d-8f7b-ca52431af32d" (UID: "16cafd0c-7aa9-498d-8f7b-ca52431af32d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.444227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.444141 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.444227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.444170 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr2pv\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-kube-api-access-rr2pv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.444227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.444185 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.444227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:34.444198 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16cafd0c-7aa9-498d-8f7b-ca52431af32d-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:35.257966 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:35.257927 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" event={"ID":"16cafd0c-7aa9-498d-8f7b-ca52431af32d","Type":"ContainerDied","Data":"83739b8053b9c10dce6fdb840fa8e2e702f44835418abf180e8e5082dedacfbf"} Nov 03 13:32:35.257966 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:35.257955 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd" Nov 03 13:32:35.258481 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:35.258005 4541 scope.go:122] "RemoveContainer" containerID="c051e2e60cdb4b5954a98215304609ed02d88caa348b209ad41cd7ebd013afde" Nov 03 13:32:35.483766 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:35.483725 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd"] Nov 03 13:32:35.530571 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:35.530491 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5vkjd"] Nov 03 13:32:36.869527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:36.869490 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" path="/var/lib/kubelet/pods/16cafd0c-7aa9-498d-8f7b-ca52431af32d/volumes" Nov 03 13:32:40.270923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.270885 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d"] Nov 03 13:32:40.271958 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.271936 4541 cpu_manager.go:414] "containerMap: removing container" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" Nov 03 13:32:40.272044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.271960 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" Nov 03 13:32:40.272130 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.272118 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="16cafd0c-7aa9-498d-8f7b-ca52431af32d" containerName="istio-proxy" Nov 03 13:32:40.277427 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.277379 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.345588 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.345552 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d"] Nov 03 13:32:40.364939 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.364901 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.365227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.365195 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.365395 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.365370 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.365463 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.365445 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.365545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.365525 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.365609 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.365593 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.365653 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.365622 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.365750 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.365730 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.365883 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.365862 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrhz\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-kube-api-access-lhrhz\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.467601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.467569 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrhz\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-kube-api-access-lhrhz\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.467800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.467628 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.467800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.467655 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.467800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.467717 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.467800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.467757 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.467993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.467816 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.467993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.467877 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.467993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.467901 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.467993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.467971 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.468245 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.468194 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.468245 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.468222 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.468375 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.468270 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.468375 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.468320 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.468561 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.468536 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.469938 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.469918 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.470324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.470307 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.525342 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.525274 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.525475 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.525452 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrhz\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-kube-api-access-lhrhz\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:40.599035 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:40.599002 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:41.021658 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:41.021621 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d"] Nov 03 13:32:41.055528 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:32:41.055488 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c33209_ae02_49e1_877d_4fbf55a18c4c.slice/crio-3db3f54930e6866713c8359df90c471490510f2f5961e18a9ad100e64d0a3d7b WatchSource:0}: Error finding container 3db3f54930e6866713c8359df90c471490510f2f5961e18a9ad100e64d0a3d7b: Status 404 returned error can't find the container with id 3db3f54930e6866713c8359df90c471490510f2f5961e18a9ad100e64d0a3d7b Nov 03 13:32:41.059439 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:41.059389 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:41.059543 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:41.059500 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:41.059587 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:41.059547 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:41.314893 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:41.314780 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" event={"ID":"a9c33209-ae02-49e1-877d-4fbf55a18c4c","Type":"ContainerStarted","Data":"fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7"} Nov 03 13:32:41.314893 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:41.314857 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" event={"ID":"a9c33209-ae02-49e1-877d-4fbf55a18c4c","Type":"ContainerStarted","Data":"3db3f54930e6866713c8359df90c471490510f2f5961e18a9ad100e64d0a3d7b"} Nov 03 13:32:41.601434 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:41.601341 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:42.601084 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:42.601052 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:32:42.601505 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:42.601105 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 13:32:43.599984 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:43.599948 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:32:43.600165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:43.600019 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 13:32:44.599905 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:44.599865 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:32:44.600309 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:44.599936 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 13:32:45.599641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:45.599583 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:32:45.599861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:45.599699 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 13:32:46.600146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:46.600109 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 03 13:32:46.600549 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:46.600171 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 03 13:32:46.891417 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:32:46.891326 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:32:47.608175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:47.608118 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:47.609563 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:47.609528 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:47.611070 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:47.611044 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:32:47.724295 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:47.724235 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" podStartSLOduration=7.724221613 podStartE2EDuration="7.724221613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:41.48408143 +0000 UTC m=+1692.798053291" watchObservedRunningTime="2026-11-03 13:32:47.724221613 +0000 UTC m=+1699.038193473" Nov 03 13:32:55.363057 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.363018 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp"] Nov 03 13:32:55.367358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.367335 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.462669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.462637 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp"] Nov 03 13:32:55.494050 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.494009 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.494377 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.494345 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.494481 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.494438 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.494536 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.494495 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.494641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.494619 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2e35f1-95b8-4488-95f4-b461b2666007-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.494818 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.494790 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cj6h\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-kube-api-access-8cj6h\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.494881 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.494852 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.494978 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.494955 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.495082 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.495065 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.596548 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.596508 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.596548 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.596550 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.596884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.596604 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.596884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.596654 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.596884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.596722 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.596884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.596796 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.596884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.596840 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.596884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.596867 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2e35f1-95b8-4488-95f4-b461b2666007-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.597280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.596991 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cj6h\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-kube-api-access-8cj6h\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.597280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.597024 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.597280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.597140 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.597280 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.597261 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.597514 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.597489 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.597581 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.597563 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2e35f1-95b8-4488-95f4-b461b2666007-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.600151 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.600132 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.600297 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.600280 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.651731 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.651702 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.652094 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.652066 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cj6h\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-kube-api-access-8cj6h\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:55.681485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:55.681448 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:56.110990 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:32:56.110956 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2e35f1_95b8_4488_95f4_b461b2666007.slice/crio-0997756a67183eebebf92811b4ab44c7369ebf8045350b83aa28df1f7376ea50 WatchSource:0}: Error finding container 0997756a67183eebebf92811b4ab44c7369ebf8045350b83aa28df1f7376ea50: Status 404 returned error can't find the container with id 0997756a67183eebebf92811b4ab44c7369ebf8045350b83aa28df1f7376ea50 Nov 03 13:32:56.112636 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:56.112612 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp"] Nov 03 13:32:56.113834 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:56.113765 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:56.113937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:56.113893 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:56.114010 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:56.113941 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:32:56.441966 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:56.441924 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" event={"ID":"bb2e35f1-95b8-4488-95f4-b461b2666007","Type":"ContainerStarted","Data":"9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037"} Nov 03 13:32:56.442430 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:56.441997 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" event={"ID":"bb2e35f1-95b8-4488-95f4-b461b2666007","Type":"ContainerStarted","Data":"0997756a67183eebebf92811b4ab44c7369ebf8045350b83aa28df1f7376ea50"} Nov 03 13:32:56.557438 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:56.557371 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" podStartSLOduration=1.557350677 podStartE2EDuration="1.557350677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:56.554617339 +0000 UTC m=+1707.868589199" watchObservedRunningTime="2026-11-03 13:32:56.557350677 +0000 UTC m=+1707.871322543" Nov 03 13:32:56.682644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:56.682600 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:32:57.684340 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:57.684308 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 13:32:57.684736 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:57.684361 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 13:32:58.682158 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:58.682122 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 13:32:58.682337 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:58.682176 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 13:32:59.682402 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:59.682370 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 13:32:59.682844 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:32:59.682423 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 13:33:00.681930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:00.681893 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 13:33:00.682252 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:00.682212 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 13:33:01.682204 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:01.682170 4541 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 03 13:33:01.682621 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:01.682226 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 03 13:33:03.154834 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:03.154798 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:33:03.156032 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:03.156007 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:33:03.157144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:03.157111 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:33:16.890289 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:33:16.890248 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:33:38.193167 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.193131 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t"] Nov 03 13:33:38.193594 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.193343 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" podUID="347b5f6e-0def-408a-9739-fceb7e2d647b" containerName="maas-api" containerID="cri-o://2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65" gracePeriod=30 Nov 03 13:33:38.508576 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.508551 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:33:38.602092 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.602049 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-maas-api-tls\") pod \"347b5f6e-0def-408a-9739-fceb7e2d647b\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " Nov 03 13:33:38.602293 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.602176 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347b5f6e-0def-408a-9739-fceb7e2d647b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347b5f6e-0def-408a-9739-fceb7e2d647b-openshift-service-ca\") pod \"347b5f6e-0def-408a-9739-fceb7e2d647b\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " Nov 03 13:33:38.602293 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.602286 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347b5f6e-0def-408a-9739-fceb7e2d647b-kube-api-access-ghnhg\" (UniqueName: \"kubernetes.io/projected/347b5f6e-0def-408a-9739-fceb7e2d647b-kube-api-access-ghnhg\") pod \"347b5f6e-0def-408a-9739-fceb7e2d647b\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " Nov 03 13:33:38.602403 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.602361 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-metrics-certs\") pod \"347b5f6e-0def-408a-9739-fceb7e2d647b\" (UID: \"347b5f6e-0def-408a-9739-fceb7e2d647b\") " Nov 03 13:33:38.603338 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.603305 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347b5f6e-0def-408a-9739-fceb7e2d647b-openshift-service-ca" pod "347b5f6e-0def-408a-9739-fceb7e2d647b" (UID: "347b5f6e-0def-408a-9739-fceb7e2d647b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:38.607473 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.607429 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-metrics-certs" pod "347b5f6e-0def-408a-9739-fceb7e2d647b" (UID: "347b5f6e-0def-408a-9739-fceb7e2d647b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:33:38.607589 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.607560 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347b5f6e-0def-408a-9739-fceb7e2d647b-kube-api-access-ghnhg" pod "347b5f6e-0def-408a-9739-fceb7e2d647b" (UID: "347b5f6e-0def-408a-9739-fceb7e2d647b"). InnerVolumeSpecName "kube-api-access-ghnhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:38.608149 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.608124 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-maas-api-tls" pod "347b5f6e-0def-408a-9739-fceb7e2d647b" (UID: "347b5f6e-0def-408a-9739-fceb7e2d647b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:33:38.704934 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.704891 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:38.704934 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.704926 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/347b5f6e-0def-408a-9739-fceb7e2d647b-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:38.705141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.704941 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/347b5f6e-0def-408a-9739-fceb7e2d647b-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:38.705141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.704956 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghnhg\" (UniqueName: \"kubernetes.io/projected/347b5f6e-0def-408a-9739-fceb7e2d647b-kube-api-access-ghnhg\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:38.721592 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.721560 4541 generic.go:356] "Generic (PLEG): container finished" podID="347b5f6e-0def-408a-9739-fceb7e2d647b" containerID="2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65" exitCode=0 Nov 03 13:33:38.721758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.721604 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" Nov 03 13:33:38.721758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.721650 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" event={"ID":"347b5f6e-0def-408a-9739-fceb7e2d647b","Type":"ContainerDied","Data":"2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65"} Nov 03 13:33:38.721758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.721704 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t" event={"ID":"347b5f6e-0def-408a-9739-fceb7e2d647b","Type":"ContainerDied","Data":"f539f2539ace2acfe040383790b5d49d01ba1e785dec64cd604057f910e07e92"} Nov 03 13:33:38.721758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.721725 4541 scope.go:122] "RemoveContainer" containerID="2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65" Nov 03 13:33:38.731911 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.731895 4541 scope.go:122] "RemoveContainer" containerID="2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65" Nov 03 13:33:38.732170 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:33:38.732149 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65\": container with ID starting with 2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65 not found: ID does not exist" containerID="2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65" Nov 03 13:33:38.732261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.732176 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65"} err="failed to get container status \"2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65\": rpc error: code = NotFound desc = could not find container \"2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65\": container with ID starting with 2020261bac1e56b1b3563e62d41b78ea34071248e70ad3c4d1683a1e886f4d65 not found: ID does not exist" Nov 03 13:33:38.878242 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.878208 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t"] Nov 03 13:33:38.881821 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:38.881761 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-28d87c-bf645f655-xfw8t"] Nov 03 13:33:40.870101 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:40.870024 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347b5f6e-0def-408a-9739-fceb7e2d647b" path="/var/lib/kubelet/pods/347b5f6e-0def-408a-9739-fceb7e2d647b/volumes" Nov 03 13:33:43.262026 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:43.261994 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r"] Nov 03 13:33:43.262448 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:43.262234 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" containerID="cri-o://b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8" gracePeriod=30 Nov 03 13:33:43.292472 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:43.292439 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv"] Nov 03 13:33:43.292703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:43.292678 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" containerID="cri-o://eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48" gracePeriod=30 Nov 03 13:33:46.890808 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:33:46.890748 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:33:48.615403 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.615374 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:33:48.618632 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.618611 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:33:48.669066 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669037 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-data\") pod \"e017540c-faef-4bae-ab8e-ca161785ad52\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " Nov 03 13:33:48.669233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669075 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e017540c-faef-4bae-ab8e-ca161785ad52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017540c-faef-4bae-ab8e-ca161785ad52-istio-podinfo\") pod \"e017540c-faef-4bae-ab8e-ca161785ad52\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " Nov 03 13:33:48.669233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669103 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-podinfo\") pod \"112a1add-c290-4f08-b1c5-48118ce06a2b\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " Nov 03 13:33:48.669233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669197 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/112a1add-c290-4f08-b1c5-48118ce06a2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112a1add-c290-4f08-b1c5-48118ce06a2b-istiod-ca-cert\") pod \"112a1add-c290-4f08-b1c5-48118ce06a2b\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " Nov 03 13:33:48.669233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669224 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-data\") pod \"112a1add-c290-4f08-b1c5-48118ce06a2b\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " Nov 03 13:33:48.669461 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669257 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-kube-api-access-5dj4q\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-kube-api-access-5dj4q\") pod \"e017540c-faef-4bae-ab8e-ca161785ad52\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " Nov 03 13:33:48.669461 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669343 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-envoy\") pod \"112a1add-c290-4f08-b1c5-48118ce06a2b\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " Nov 03 13:33:48.669461 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669376 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-credential-socket\") pod \"e017540c-faef-4bae-ab8e-ca161785ad52\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " Nov 03 13:33:48.669461 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669385 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-data" pod "e017540c-faef-4bae-ab8e-ca161785ad52" (UID: "e017540c-faef-4bae-ab8e-ca161785ad52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.669461 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669401 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e017540c-faef-4bae-ab8e-ca161785ad52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017540c-faef-4bae-ab8e-ca161785ad52-istiod-ca-cert\") pod \"e017540c-faef-4bae-ab8e-ca161785ad52\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " Nov 03 13:33:48.669461 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669448 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-certs\") pod \"112a1add-c290-4f08-b1c5-48118ce06a2b\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " Nov 03 13:33:48.669761 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669560 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-envoy\") pod \"e017540c-faef-4bae-ab8e-ca161785ad52\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " Nov 03 13:33:48.669761 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669589 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-certs\") pod \"e017540c-faef-4bae-ab8e-ca161785ad52\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " Nov 03 13:33:48.669761 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669625 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-credential-socket\") pod \"112a1add-c290-4f08-b1c5-48118ce06a2b\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " Nov 03 13:33:48.669761 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669630 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-credential-socket" pod "e017540c-faef-4bae-ab8e-ca161785ad52" (UID: "e017540c-faef-4bae-ab8e-ca161785ad52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.669761 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669632 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-data" pod "112a1add-c290-4f08-b1c5-48118ce06a2b" (UID: "112a1add-c290-4f08-b1c5-48118ce06a2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.669761 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669744 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-kube-api-access-p7s8j\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-kube-api-access-p7s8j\") pod \"112a1add-c290-4f08-b1c5-48118ce06a2b\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " Nov 03 13:33:48.670131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669857 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-istio-token\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-istio-token\") pod \"e017540c-faef-4bae-ab8e-ca161785ad52\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " Nov 03 13:33:48.670131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669949 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112a1add-c290-4f08-b1c5-48118ce06a2b-istiod-ca-cert" pod "112a1add-c290-4f08-b1c5-48118ce06a2b" (UID: "112a1add-c290-4f08-b1c5-48118ce06a2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:48.670131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.669990 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-token\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-token\") pod \"112a1add-c290-4f08-b1c5-48118ce06a2b\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " Nov 03 13:33:48.670131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670024 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-socket\") pod \"112a1add-c290-4f08-b1c5-48118ce06a2b\" (UID: \"112a1add-c290-4f08-b1c5-48118ce06a2b\") " Nov 03 13:33:48.670131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670053 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-socket\") pod \"e017540c-faef-4bae-ab8e-ca161785ad52\" (UID: \"e017540c-faef-4bae-ab8e-ca161785ad52\") " Nov 03 13:33:48.670358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670129 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-credential-socket" pod "112a1add-c290-4f08-b1c5-48118ce06a2b" (UID: "112a1add-c290-4f08-b1c5-48118ce06a2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.670431 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670411 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e017540c-faef-4bae-ab8e-ca161785ad52-istiod-ca-cert" pod "e017540c-faef-4bae-ab8e-ca161785ad52" (UID: "e017540c-faef-4bae-ab8e-ca161785ad52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:48.670608 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670584 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-socket" pod "e017540c-faef-4bae-ab8e-ca161785ad52" (UID: "e017540c-faef-4bae-ab8e-ca161785ad52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.670865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670841 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.670959 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670868 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.670959 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670882 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.670959 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670896 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112a1add-c290-4f08-b1c5-48118ce06a2b-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.670959 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670909 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.670959 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670948 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.671286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.670962 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e017540c-faef-4bae-ab8e-ca161785ad52-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.671286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.671141 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-certs" pod "112a1add-c290-4f08-b1c5-48118ce06a2b" (UID: "112a1add-c290-4f08-b1c5-48118ce06a2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.672048 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.672019 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-kube-api-access-5dj4q" pod "e017540c-faef-4bae-ab8e-ca161785ad52" (UID: "e017540c-faef-4bae-ab8e-ca161785ad52"). InnerVolumeSpecName "kube-api-access-5dj4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.672586 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.672230 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-podinfo" pod "112a1add-c290-4f08-b1c5-48118ce06a2b" (UID: "112a1add-c290-4f08-b1c5-48118ce06a2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:48.672586 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.672345 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e017540c-faef-4bae-ab8e-ca161785ad52-istio-podinfo" pod "e017540c-faef-4bae-ab8e-ca161785ad52" (UID: "e017540c-faef-4bae-ab8e-ca161785ad52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:48.672586 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.672500 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-kube-api-access-p7s8j" pod "112a1add-c290-4f08-b1c5-48118ce06a2b" (UID: "112a1add-c290-4f08-b1c5-48118ce06a2b"). InnerVolumeSpecName "kube-api-access-p7s8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.672586 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.672534 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-certs" pod "e017540c-faef-4bae-ab8e-ca161785ad52" (UID: "e017540c-faef-4bae-ab8e-ca161785ad52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.672879 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.672596 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-socket" pod "112a1add-c290-4f08-b1c5-48118ce06a2b" (UID: "112a1add-c290-4f08-b1c5-48118ce06a2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.672879 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.672729 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-istio-token" pod "e017540c-faef-4bae-ab8e-ca161785ad52" (UID: "e017540c-faef-4bae-ab8e-ca161785ad52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.673369 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.673346 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-envoy" pod "e017540c-faef-4bae-ab8e-ca161785ad52" (UID: "e017540c-faef-4bae-ab8e-ca161785ad52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.673452 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.673388 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-envoy" pod "112a1add-c290-4f08-b1c5-48118ce06a2b" (UID: "112a1add-c290-4f08-b1c5-48118ce06a2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.674007 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.673988 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-token" pod "112a1add-c290-4f08-b1c5-48118ce06a2b" (UID: "112a1add-c290-4f08-b1c5-48118ce06a2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.772900 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772858 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.772900 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772891 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.772900 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772908 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.773150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772922 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e017540c-faef-4bae-ab8e-ca161785ad52-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.773150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772935 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7s8j\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-kube-api-access-p7s8j\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.773150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772949 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.773150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772962 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.773150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772975 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112a1add-c290-4f08-b1c5-48118ce06a2b-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.773150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772987 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e017540c-faef-4bae-ab8e-ca161785ad52-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.773150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.772998 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112a1add-c290-4f08-b1c5-48118ce06a2b-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.773150 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.773012 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dj4q\" (UniqueName: \"kubernetes.io/projected/e017540c-faef-4bae-ab8e-ca161785ad52-kube-api-access-5dj4q\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.788163 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.788131 4541 generic.go:356] "Generic (PLEG): container finished" podID="e017540c-faef-4bae-ab8e-ca161785ad52" containerID="eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48" exitCode=0 Nov 03 13:33:48.788358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.788208 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" Nov 03 13:33:48.788358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.788223 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" event={"ID":"e017540c-faef-4bae-ab8e-ca161785ad52","Type":"ContainerDied","Data":"eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48"} Nov 03 13:33:48.788358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.788320 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv" event={"ID":"e017540c-faef-4bae-ab8e-ca161785ad52","Type":"ContainerDied","Data":"dd144e74e146ffffb15bdba2e96e90950b0b9fd04e4aad757f8c7c69ed3dcc41"} Nov 03 13:33:48.788358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.788351 4541 scope.go:122] "RemoveContainer" containerID="eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48" Nov 03 13:33:48.789909 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.789871 4541 generic.go:356] "Generic (PLEG): container finished" podID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerID="b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8" exitCode=0 Nov 03 13:33:48.789986 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.789931 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" Nov 03 13:33:48.789986 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.789966 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" event={"ID":"112a1add-c290-4f08-b1c5-48118ce06a2b","Type":"ContainerDied","Data":"b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8"} Nov 03 13:33:48.790071 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.790030 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r" event={"ID":"112a1add-c290-4f08-b1c5-48118ce06a2b","Type":"ContainerDied","Data":"9ce0dc0c9614472bb746bb4af267ccaec1f24fbe23cc3d88e3f8cd2738c346d0"} Nov 03 13:33:48.802039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.800919 4541 scope.go:122] "RemoveContainer" containerID="eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48" Nov 03 13:33:48.802039 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:33:48.801239 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48\": container with ID starting with eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48 not found: ID does not exist" containerID="eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48" Nov 03 13:33:48.802039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.801272 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48"} err="failed to get container status \"eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48\": rpc error: code = NotFound desc = could not find container \"eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48\": container with ID starting with eb7c42bfed8d6358af74abd9e90b478e56fe9b77088b37563acee4c6e9afbc48 not found: ID does not exist" Nov 03 13:33:48.802039 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.801299 4541 scope.go:122] "RemoveContainer" containerID="b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8" Nov 03 13:33:48.820628 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.820604 4541 scope.go:122] "RemoveContainer" containerID="b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8" Nov 03 13:33:48.821012 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:33:48.820990 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8\": container with ID starting with b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8 not found: ID does not exist" containerID="b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8" Nov 03 13:33:48.821104 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:48.821019 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8"} err="failed to get container status \"b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8\": rpc error: code = NotFound desc = could not find container \"b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8\": container with ID starting with b61d8255fc58a05e23ea96a545b10216e00114cf6795889d80eca9ae7e6aa8a8 not found: ID does not exist" Nov 03 13:33:49.003472 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:49.003441 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r"] Nov 03 13:33:49.006587 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:49.006542 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-zj29r"] Nov 03 13:33:49.111533 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:49.111452 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv"] Nov 03 13:33:49.120005 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:49.119974 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-mgxvv"] Nov 03 13:33:50.881287 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:50.881253 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" path="/var/lib/kubelet/pods/112a1add-c290-4f08-b1c5-48118ce06a2b/volumes" Nov 03 13:33:50.881805 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:33:50.881788 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" path="/var/lib/kubelet/pods/e017540c-faef-4bae-ab8e-ca161785ad52/volumes" Nov 03 13:34:16.898615 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:34:16.898576 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:34:29.085630 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:34:29.085597 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:34:29.086920 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:34:29.086895 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:34:46.890199 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:34:46.890162 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:35:04.664197 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:04.664161 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp"] Nov 03 13:35:04.664795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:04.664364 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerName="istio-proxy" containerID="cri-o://9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037" gracePeriod=30 Nov 03 13:35:04.672958 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:04.672929 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d"] Nov 03 13:35:04.673187 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:04.673164 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerName="istio-proxy" containerID="cri-o://fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7" gracePeriod=30 Nov 03 13:35:04.686847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:04.686811 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8"] Nov 03 13:35:04.687190 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:04.687135 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" containerID="cri-o://a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8" gracePeriod=30 Nov 03 13:35:09.995841 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:09.995810 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:35:09.998156 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:09.998133 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:35:10.036334 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.036310 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:35:10.079896 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.079859 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-data\") pod \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " Nov 03 13:35:10.080062 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.079948 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istiod-ca-cert\") pod \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " Nov 03 13:35:10.080062 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.079972 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-certs\") pod \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " Nov 03 13:35:10.080062 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080008 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-envoy\") pod \"bb2e35f1-95b8-4488-95f4-b461b2666007\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " Nov 03 13:35:10.080062 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080055 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-envoy\") pod \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " Nov 03 13:35:10.080286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080078 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-token\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-token\") pod \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " Nov 03 13:35:10.080286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080102 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-certs\") pod \"bb2e35f1-95b8-4488-95f4-b461b2666007\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " Nov 03 13:35:10.080286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080142 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-podinfo\") pod \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " Nov 03 13:35:10.080286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080139 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-data" pod "a9c33209-ae02-49e1-877d-4fbf55a18c4c" (UID: "a9c33209-ae02-49e1-877d-4fbf55a18c4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.080286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080170 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-credential-socket\") pod \"bb2e35f1-95b8-4488-95f4-b461b2666007\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " Nov 03 13:35:10.080545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080329 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-socket\") pod \"bb2e35f1-95b8-4488-95f4-b461b2666007\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " Nov 03 13:35:10.080545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080375 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-certs" pod "bb2e35f1-95b8-4488-95f4-b461b2666007" (UID: "bb2e35f1-95b8-4488-95f4-b461b2666007"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.080545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080388 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-credential-socket" pod "bb2e35f1-95b8-4488-95f4-b461b2666007" (UID: "bb2e35f1-95b8-4488-95f4-b461b2666007"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.080545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080390 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-kube-api-access-8cj6h\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-kube-api-access-8cj6h\") pod \"bb2e35f1-95b8-4488-95f4-b461b2666007\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " Nov 03 13:35:10.080545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080416 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istiod-ca-cert" pod "a9c33209-ae02-49e1-877d-4fbf55a18c4c" (UID: "a9c33209-ae02-49e1-877d-4fbf55a18c4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.080545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080521 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-kube-api-access-lhrhz\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-kube-api-access-lhrhz\") pod \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " Nov 03 13:35:10.080887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080601 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-socket\") pod \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " Nov 03 13:35:10.080887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080608 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-socket" pod "bb2e35f1-95b8-4488-95f4-b461b2666007" (UID: "bb2e35f1-95b8-4488-95f4-b461b2666007"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.080887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080623 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-certs" pod "a9c33209-ae02-49e1-877d-4fbf55a18c4c" (UID: "a9c33209-ae02-49e1-877d-4fbf55a18c4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.080887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080634 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-token\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-token\") pod \"bb2e35f1-95b8-4488-95f4-b461b2666007\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " Nov 03 13:35:10.080887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080721 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-data\") pod \"bb2e35f1-95b8-4488-95f4-b461b2666007\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " Nov 03 13:35:10.080887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080764 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-podinfo\") pod \"bb2e35f1-95b8-4488-95f4-b461b2666007\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " Nov 03 13:35:10.080887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080839 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-credential-socket\") pod \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\" (UID: \"a9c33209-ae02-49e1-877d-4fbf55a18c4c\") " Nov 03 13:35:10.080887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080872 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-socket" pod "a9c33209-ae02-49e1-877d-4fbf55a18c4c" (UID: "a9c33209-ae02-49e1-877d-4fbf55a18c4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.081305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.080903 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb2e35f1-95b8-4488-95f4-b461b2666007-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2e35f1-95b8-4488-95f4-b461b2666007-istiod-ca-cert\") pod \"bb2e35f1-95b8-4488-95f4-b461b2666007\" (UID: \"bb2e35f1-95b8-4488-95f4-b461b2666007\") " Nov 03 13:35:10.081473 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081430 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-credential-socket" pod "a9c33209-ae02-49e1-877d-4fbf55a18c4c" (UID: "a9c33209-ae02-49e1-877d-4fbf55a18c4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.081473 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081438 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-data" pod "bb2e35f1-95b8-4488-95f4-b461b2666007" (UID: "bb2e35f1-95b8-4488-95f4-b461b2666007"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.081687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081550 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.081687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081570 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.081687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081587 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.081687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081602 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.081687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081617 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.081687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081630 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.081687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081645 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.081687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081658 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.081687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081671 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.082229 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.081687 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb2e35f1-95b8-4488-95f4-b461b2666007-istiod-ca-cert" pod "bb2e35f1-95b8-4488-95f4-b461b2666007" (UID: "bb2e35f1-95b8-4488-95f4-b461b2666007"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.083082 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.083044 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-envoy" pod "a9c33209-ae02-49e1-877d-4fbf55a18c4c" (UID: "a9c33209-ae02-49e1-877d-4fbf55a18c4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.083314 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.083286 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-token" pod "bb2e35f1-95b8-4488-95f4-b461b2666007" (UID: "bb2e35f1-95b8-4488-95f4-b461b2666007"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.083469 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.083444 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-kube-api-access-8cj6h" pod "bb2e35f1-95b8-4488-95f4-b461b2666007" (UID: "bb2e35f1-95b8-4488-95f4-b461b2666007"). InnerVolumeSpecName "kube-api-access-8cj6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.083552 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.083448 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-token" pod "a9c33209-ae02-49e1-877d-4fbf55a18c4c" (UID: "a9c33209-ae02-49e1-877d-4fbf55a18c4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.083552 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.083485 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-podinfo" pod "a9c33209-ae02-49e1-877d-4fbf55a18c4c" (UID: "a9c33209-ae02-49e1-877d-4fbf55a18c4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.083724 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.083705 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-envoy" pod "bb2e35f1-95b8-4488-95f4-b461b2666007" (UID: "bb2e35f1-95b8-4488-95f4-b461b2666007"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.083984 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.083967 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-kube-api-access-lhrhz" pod "a9c33209-ae02-49e1-877d-4fbf55a18c4c" (UID: "a9c33209-ae02-49e1-877d-4fbf55a18c4c"). InnerVolumeSpecName "kube-api-access-lhrhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.084372 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.084352 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-podinfo" pod "bb2e35f1-95b8-4488-95f4-b461b2666007" (UID: "bb2e35f1-95b8-4488-95f4-b461b2666007"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.183327 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.183288 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-credential-socket\") pod \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " Nov 03 13:35:10.183573 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.183355 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-token\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-token\") pod \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " Nov 03 13:35:10.183573 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.183403 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-podinfo\") pod \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " Nov 03 13:35:10.183573 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.183535 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-certs\") pod \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " Nov 03 13:35:10.184336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.183599 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-socket\") pod \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " Nov 03 13:35:10.184336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.183655 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-kube-api-access-n7jwf\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-kube-api-access-n7jwf\") pod \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " Nov 03 13:35:10.184336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.183706 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-envoy\") pod \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " Nov 03 13:35:10.184336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.183761 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-data\") pod \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " Nov 03 13:35:10.184336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.183821 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istiod-ca-cert\") pod \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\" (UID: \"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5\") " Nov 03 13:35:10.184648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184531 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-credential-socket" pod "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" (UID: "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.184648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184543 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.184648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184591 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb2e35f1-95b8-4488-95f4-b461b2666007-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.184648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184609 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.184648 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184647 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.184937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184662 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.184937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184675 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9c33209-ae02-49e1-877d-4fbf55a18c4c-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.184937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184691 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cj6h\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-kube-api-access-8cj6h\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.184937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184708 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrhz\" (UniqueName: \"kubernetes.io/projected/a9c33209-ae02-49e1-877d-4fbf55a18c4c-kube-api-access-lhrhz\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.184937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184721 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb2e35f1-95b8-4488-95f4-b461b2666007-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.184937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184764 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-certs" pod "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" (UID: "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.184937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184821 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-socket" pod "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" (UID: "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.184937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184860 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-data" pod "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" (UID: "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.184937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.184890 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istiod-ca-cert" pod "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" (UID: "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.187265 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.187241 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-envoy" pod "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" (UID: "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.187547 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.187425 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-podinfo" pod "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" (UID: "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.187547 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.187514 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-kube-api-access-n7jwf" pod "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" (UID: "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5"). InnerVolumeSpecName "kube-api-access-n7jwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.187685 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.187588 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-token" pod "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" (UID: "6ef1e1f4-063c-4daa-8a31-f40d7bd561d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.286194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.286112 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.286194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.286141 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7jwf\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-kube-api-access-n7jwf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.286194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.286152 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.286194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.286162 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.286194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.286171 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.286194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.286181 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.286194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.286189 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.286194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.286198 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.286561 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.286206 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.378188 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.378146 4541 generic.go:356] "Generic (PLEG): container finished" podID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerID="9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037" exitCode=0 Nov 03 13:35:10.378392 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.378197 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" Nov 03 13:35:10.378392 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.378236 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" event={"ID":"bb2e35f1-95b8-4488-95f4-b461b2666007","Type":"ContainerDied","Data":"9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037"} Nov 03 13:35:10.378392 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.378299 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp" event={"ID":"bb2e35f1-95b8-4488-95f4-b461b2666007","Type":"ContainerDied","Data":"0997756a67183eebebf92811b4ab44c7369ebf8045350b83aa28df1f7376ea50"} Nov 03 13:35:10.378392 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.378329 4541 scope.go:122] "RemoveContainer" containerID="9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037" Nov 03 13:35:10.379949 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.379924 4541 generic.go:356] "Generic (PLEG): container finished" podID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerID="a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8" exitCode=0 Nov 03 13:35:10.380051 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.379964 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" Nov 03 13:35:10.380051 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.380014 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" event={"ID":"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5","Type":"ContainerDied","Data":"a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8"} Nov 03 13:35:10.380163 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.380068 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8" event={"ID":"6ef1e1f4-063c-4daa-8a31-f40d7bd561d5","Type":"ContainerDied","Data":"549e6b26be5076b28553f593e141809c5801aef21595bb16f268bbdd782f3a2f"} Nov 03 13:35:10.381651 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.381624 4541 generic.go:356] "Generic (PLEG): container finished" podID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerID="fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7" exitCode=0 Nov 03 13:35:10.381813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.381653 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" event={"ID":"a9c33209-ae02-49e1-877d-4fbf55a18c4c","Type":"ContainerDied","Data":"fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7"} Nov 03 13:35:10.381813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.381681 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" Nov 03 13:35:10.381813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.381701 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d" event={"ID":"a9c33209-ae02-49e1-877d-4fbf55a18c4c","Type":"ContainerDied","Data":"3db3f54930e6866713c8359df90c471490510f2f5961e18a9ad100e64d0a3d7b"} Nov 03 13:35:10.389377 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.389353 4541 scope.go:122] "RemoveContainer" containerID="9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037" Nov 03 13:35:10.389659 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:35:10.389630 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037\": container with ID starting with 9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037 not found: ID does not exist" containerID="9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037" Nov 03 13:35:10.389761 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.389669 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037"} err="failed to get container status \"9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037\": rpc error: code = NotFound desc = could not find container \"9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037\": container with ID starting with 9d82520d62c65731f7247d68896807f86551680da942c847852e2a79c7e72037 not found: ID does not exist" Nov 03 13:35:10.389761 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.389696 4541 scope.go:122] "RemoveContainer" containerID="a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8" Nov 03 13:35:10.398907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.398885 4541 scope.go:122] "RemoveContainer" containerID="a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8" Nov 03 13:35:10.399176 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:35:10.399157 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8\": container with ID starting with a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8 not found: ID does not exist" containerID="a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8" Nov 03 13:35:10.399263 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.399188 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8"} err="failed to get container status \"a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8\": rpc error: code = NotFound desc = could not find container \"a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8\": container with ID starting with a5afef0bf5f05e7079b8d094962c18da933dc9c3cd614095fa089aa07340a9e8 not found: ID does not exist" Nov 03 13:35:10.399263 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.399215 4541 scope.go:122] "RemoveContainer" containerID="fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7" Nov 03 13:35:10.408817 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.408759 4541 scope.go:122] "RemoveContainer" containerID="fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7" Nov 03 13:35:10.409106 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:35:10.409087 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7\": container with ID starting with fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7 not found: ID does not exist" containerID="fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7" Nov 03 13:35:10.409168 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.409111 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7"} err="failed to get container status \"fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7\": rpc error: code = NotFound desc = could not find container \"fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7\": container with ID starting with fb88a0bff059a7fb81b8a26b2d2ce520ee7d7e6a52b90518a17d694cfa2905a7 not found: ID does not exist" Nov 03 13:35:10.543071 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.542754 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8"] Nov 03 13:35:10.546051 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.546019 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-cbmx8"] Nov 03 13:35:10.648848 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.648810 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d"] Nov 03 13:35:10.656447 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.656412 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-xf47d"] Nov 03 13:35:10.765866 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.765827 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp"] Nov 03 13:35:10.767873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.767847 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-l4slp"] Nov 03 13:35:10.870709 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.870629 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" path="/var/lib/kubelet/pods/6ef1e1f4-063c-4daa-8a31-f40d7bd561d5/volumes" Nov 03 13:35:10.871176 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.871159 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" path="/var/lib/kubelet/pods/a9c33209-ae02-49e1-877d-4fbf55a18c4c/volumes" Nov 03 13:35:10.871623 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:35:10.871608 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" path="/var/lib/kubelet/pods/bb2e35f1-95b8-4488-95f4-b461b2666007/volumes" Nov 03 13:35:16.890371 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:35:16.890331 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:35:46.890596 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:35:46.890558 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:36:16.889870 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:36:16.889831 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:36:24.953808 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:24.953734 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd"] Nov 03 13:36:24.954411 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:24.953959 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" containerID="cri-o://04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98" gracePeriod=30 Nov 03 13:36:26.066873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:26.066831 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt"] Nov 03 13:36:26.067351 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:26.067076 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" containerID="cri-o://085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc" gracePeriod=30 Nov 03 13:36:26.667139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:26.667098 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:36:26.667313 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:26.667173 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:36:29.009626 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:29.009582 4541 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:36:29.010112 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:29.009663 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:36:30.216352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.216326 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:36:30.393100 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393016 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istiod-ca-cert\") pod \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " Nov 03 13:36:30.393100 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393089 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-podinfo\") pod \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " Nov 03 13:36:30.393317 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393219 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-kube-api-access-qvndg\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-kube-api-access-qvndg\") pod \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " Nov 03 13:36:30.393317 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393265 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-envoy\") pod \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " Nov 03 13:36:30.393429 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393332 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-certs\") pod \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " Nov 03 13:36:30.393484 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393424 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-credential-socket\") pod \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " Nov 03 13:36:30.393484 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393458 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-token\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-token\") pod \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " Nov 03 13:36:30.393592 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393511 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-socket\") pod \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " Nov 03 13:36:30.393649 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393597 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-data\") pod \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\" (UID: \"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8\") " Nov 03 13:36:30.393703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393642 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-certs" pod "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" (UID: "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:30.393818 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393784 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-credential-socket" pod "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" (UID: "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:30.393926 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393903 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-socket" pod "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" (UID: "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:30.394014 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.393992 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-data" pod "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" (UID: "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:30.394071 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.394029 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istiod-ca-cert" pod "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" (UID: "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:30.394217 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.394201 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:30.394291 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.394218 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:30.394291 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.394227 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:30.394291 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.394237 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:30.394291 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.394248 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:30.396793 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.396196 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-envoy" pod "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" (UID: "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:30.397930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.397880 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-podinfo" pod "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" (UID: "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:36:30.397930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.397919 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-kube-api-access-qvndg" pod "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" (UID: "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8"). InnerVolumeSpecName "kube-api-access-qvndg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:30.398151 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.398125 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-token" pod "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" (UID: "11f98690-9c1c-462c-b1d2-a9d4a5ef11b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:30.496258 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.496223 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:30.496258 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.496260 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvndg\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-kube-api-access-qvndg\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:30.496526 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.496283 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:30.496526 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.496295 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:30.971832 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.971793 4541 generic.go:356] "Generic (PLEG): container finished" podID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerID="04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98" exitCode=0 Nov 03 13:36:30.972027 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.971861 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" Nov 03 13:36:30.972027 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.971856 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" event={"ID":"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8","Type":"ContainerDied","Data":"04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98"} Nov 03 13:36:30.972027 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.971980 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd" event={"ID":"11f98690-9c1c-462c-b1d2-a9d4a5ef11b8","Type":"ContainerDied","Data":"074538f62cca09f2bab3a3a2db67fb54aed994ee60d4e52af125a67ee9fdd338"} Nov 03 13:36:30.972027 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.972001 4541 scope.go:122] "RemoveContainer" containerID="04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98" Nov 03 13:36:30.981405 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.981387 4541 scope.go:122] "RemoveContainer" containerID="04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98" Nov 03 13:36:30.981666 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:36:30.981648 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98\": container with ID starting with 04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98 not found: ID does not exist" containerID="04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98" Nov 03 13:36:30.981711 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:30.981674 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98"} err="failed to get container status \"04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98\": rpc error: code = NotFound desc = could not find container \"04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98\": container with ID starting with 04d50b0e2e2e3651e8775dfb9548e05acb9ff2b8af66efc66b76bc0d5b020d98 not found: ID does not exist" Nov 03 13:36:31.129666 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.129581 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd"] Nov 03 13:36:31.176098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.176060 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-lzbsd"] Nov 03 13:36:31.331411 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.331385 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:36:31.407185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407155 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-credential-socket\") pod \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " Nov 03 13:36:31.407360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407217 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-socket\") pod \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " Nov 03 13:36:31.407360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407306 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-data\") pod \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " Nov 03 13:36:31.407360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407338 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istiod-ca-cert\") pod \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " Nov 03 13:36:31.407527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407388 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-token\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-token\") pod \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " Nov 03 13:36:31.407527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407431 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-certs\") pod \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " Nov 03 13:36:31.407527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407472 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-credential-socket" pod "37545439-ddd2-40d6-bf2f-a2b8c926ad5e" (UID: "37545439-ddd2-40d6-bf2f-a2b8c926ad5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:31.407527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407508 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-envoy\") pod \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " Nov 03 13:36:31.407719 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407546 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-podinfo\") pod \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " Nov 03 13:36:31.407719 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407586 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-kube-api-access-w2rvp\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-kube-api-access-w2rvp\") pod \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\" (UID: \"37545439-ddd2-40d6-bf2f-a2b8c926ad5e\") " Nov 03 13:36:31.407719 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407581 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-socket" pod "37545439-ddd2-40d6-bf2f-a2b8c926ad5e" (UID: "37545439-ddd2-40d6-bf2f-a2b8c926ad5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:31.407719 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407653 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-data" pod "37545439-ddd2-40d6-bf2f-a2b8c926ad5e" (UID: "37545439-ddd2-40d6-bf2f-a2b8c926ad5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:31.407953 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407745 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istiod-ca-cert" pod "37545439-ddd2-40d6-bf2f-a2b8c926ad5e" (UID: "37545439-ddd2-40d6-bf2f-a2b8c926ad5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:31.407953 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.407743 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-certs" pod "37545439-ddd2-40d6-bf2f-a2b8c926ad5e" (UID: "37545439-ddd2-40d6-bf2f-a2b8c926ad5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:31.408099 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.408078 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.408166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.408105 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.408166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.408121 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.408166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.408135 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.408166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.408149 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.409752 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.409722 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-token" pod "37545439-ddd2-40d6-bf2f-a2b8c926ad5e" (UID: "37545439-ddd2-40d6-bf2f-a2b8c926ad5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:31.409905 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.409758 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-podinfo" pod "37545439-ddd2-40d6-bf2f-a2b8c926ad5e" (UID: "37545439-ddd2-40d6-bf2f-a2b8c926ad5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:36:31.409905 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.409825 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-envoy" pod "37545439-ddd2-40d6-bf2f-a2b8c926ad5e" (UID: "37545439-ddd2-40d6-bf2f-a2b8c926ad5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:31.409905 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.409872 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-kube-api-access-w2rvp" pod "37545439-ddd2-40d6-bf2f-a2b8c926ad5e" (UID: "37545439-ddd2-40d6-bf2f-a2b8c926ad5e"). InnerVolumeSpecName "kube-api-access-w2rvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:31.508886 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.508855 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.508886 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.508881 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.508886 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.508891 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.509115 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.508900 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rvp\" (UniqueName: \"kubernetes.io/projected/37545439-ddd2-40d6-bf2f-a2b8c926ad5e-kube-api-access-w2rvp\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.979433 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.979399 4541 generic.go:356] "Generic (PLEG): container finished" podID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerID="085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc" exitCode=0 Nov 03 13:36:31.979622 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.979454 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" Nov 03 13:36:31.979622 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.979489 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" event={"ID":"37545439-ddd2-40d6-bf2f-a2b8c926ad5e","Type":"ContainerDied","Data":"085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc"} Nov 03 13:36:31.979622 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.979550 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt" event={"ID":"37545439-ddd2-40d6-bf2f-a2b8c926ad5e","Type":"ContainerDied","Data":"5a447d83036e16e4afb941c1f19a661241c34c0054889b86a10e727a216847a9"} Nov 03 13:36:31.979622 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.979566 4541 scope.go:122] "RemoveContainer" containerID="085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc" Nov 03 13:36:31.992616 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.992592 4541 scope.go:122] "RemoveContainer" containerID="085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc" Nov 03 13:36:31.992963 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:36:31.992929 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc\": container with ID starting with 085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc not found: ID does not exist" containerID="085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc" Nov 03 13:36:31.993076 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:31.992978 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc"} err="failed to get container status \"085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc\": rpc error: code = NotFound desc = could not find container \"085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc\": container with ID starting with 085d5b2fb50f917705640ea9570d218f23f55a9f99299cd6da89138fc2844afc not found: ID does not exist" Nov 03 13:36:32.142629 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:32.142593 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt"] Nov 03 13:36:32.190626 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:32.190576 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-8jvxt"] Nov 03 13:36:32.868703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:32.868666 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" path="/var/lib/kubelet/pods/11f98690-9c1c-462c-b1d2-a9d4a5ef11b8/volumes" Nov 03 13:36:32.869166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:32.869150 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" path="/var/lib/kubelet/pods/37545439-ddd2-40d6-bf2f-a2b8c926ad5e/volumes" Nov 03 13:36:40.079312 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:40.079220 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c"] Nov 03 13:36:40.079811 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:40.079424 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" containerID="cri-o://e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8" gracePeriod=30 Nov 03 13:36:43.300983 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:43.300947 4541 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:36:43.301355 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:43.301010 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:36:45.354304 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.354245 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:36:45.488123 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488084 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-data\") pod \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " Nov 03 13:36:45.488305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488146 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-envoy\") pod \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " Nov 03 13:36:45.488305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488182 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istiod-ca-cert\") pod \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " Nov 03 13:36:45.488305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488252 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-podinfo\") pod \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " Nov 03 13:36:45.488305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488298 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-token\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-token\") pod \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " Nov 03 13:36:45.488507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488340 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-socket\") pod \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " Nov 03 13:36:45.488507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488390 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-kube-api-access-mk557\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-kube-api-access-mk557\") pod \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " Nov 03 13:36:45.488507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488412 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-data" pod "86bbf92d-72fe-49d6-9e1b-6265bdf1b809" (UID: "86bbf92d-72fe-49d6-9e1b-6265bdf1b809"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:45.488507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488474 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-credential-socket\") pod \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " Nov 03 13:36:45.488720 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488526 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-certs\") pod \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\" (UID: \"86bbf92d-72fe-49d6-9e1b-6265bdf1b809\") " Nov 03 13:36:45.488720 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488609 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istiod-ca-cert" pod "86bbf92d-72fe-49d6-9e1b-6265bdf1b809" (UID: "86bbf92d-72fe-49d6-9e1b-6265bdf1b809"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:45.488869 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488833 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-credential-socket" pod "86bbf92d-72fe-49d6-9e1b-6265bdf1b809" (UID: "86bbf92d-72fe-49d6-9e1b-6265bdf1b809"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:45.489031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.488997 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-certs" pod "86bbf92d-72fe-49d6-9e1b-6265bdf1b809" (UID: "86bbf92d-72fe-49d6-9e1b-6265bdf1b809"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:45.489031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.489010 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-socket" pod "86bbf92d-72fe-49d6-9e1b-6265bdf1b809" (UID: "86bbf92d-72fe-49d6-9e1b-6265bdf1b809"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:45.489347 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.489322 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:45.489412 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.489353 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:45.489412 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.489369 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:45.489412 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.489384 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:45.489412 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.489397 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:45.490554 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.490529 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-envoy" pod "86bbf92d-72fe-49d6-9e1b-6265bdf1b809" (UID: "86bbf92d-72fe-49d6-9e1b-6265bdf1b809"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:45.490554 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.490542 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-kube-api-access-mk557" pod "86bbf92d-72fe-49d6-9e1b-6265bdf1b809" (UID: "86bbf92d-72fe-49d6-9e1b-6265bdf1b809"). InnerVolumeSpecName "kube-api-access-mk557". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:45.490684 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.490663 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-token" pod "86bbf92d-72fe-49d6-9e1b-6265bdf1b809" (UID: "86bbf92d-72fe-49d6-9e1b-6265bdf1b809"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:45.490684 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.490669 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-podinfo" pod "86bbf92d-72fe-49d6-9e1b-6265bdf1b809" (UID: "86bbf92d-72fe-49d6-9e1b-6265bdf1b809"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:36:45.590669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.590631 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:45.590669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.590665 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:45.590669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.590677 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:45.590921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:45.590686 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk557\" (UniqueName: \"kubernetes.io/projected/86bbf92d-72fe-49d6-9e1b-6265bdf1b809-kube-api-access-mk557\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:46.075385 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.075343 4541 generic.go:356] "Generic (PLEG): container finished" podID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerID="e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8" exitCode=0 Nov 03 13:36:46.075563 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.075382 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" event={"ID":"86bbf92d-72fe-49d6-9e1b-6265bdf1b809","Type":"ContainerDied","Data":"e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8"} Nov 03 13:36:46.075563 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.075450 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" event={"ID":"86bbf92d-72fe-49d6-9e1b-6265bdf1b809","Type":"ContainerDied","Data":"8b14ddf4dd2b708a32419d81eab7c7b83b4d7ac0f248160d5bd4d615ade173c8"} Nov 03 13:36:46.075563 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.075475 4541 scope.go:122] "RemoveContainer" containerID="e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8" Nov 03 13:36:46.075563 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.075392 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c" Nov 03 13:36:46.085813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.085792 4541 scope.go:122] "RemoveContainer" containerID="e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8" Nov 03 13:36:46.086116 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:36:46.086087 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8\": container with ID starting with e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8 not found: ID does not exist" containerID="e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8" Nov 03 13:36:46.086165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.086126 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8"} err="failed to get container status \"e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8\": rpc error: code = NotFound desc = could not find container \"e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8\": container with ID starting with e7b369288845b24e4fc9f15d1229c084e109548ed85fb1299915a7ee8a735fe8 not found: ID does not exist" Nov 03 13:36:46.279642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.279603 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c"] Nov 03 13:36:46.285239 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.285212 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-rq96c"] Nov 03 13:36:46.868899 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:36:46.868861 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" path="/var/lib/kubelet/pods/86bbf92d-72fe-49d6-9e1b-6265bdf1b809/volumes" Nov 03 13:36:46.894306 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:36:46.894263 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:37:16.891793 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:37:16.891672 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:37:31.584602 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:31.584566 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29"] Nov 03 13:37:31.585247 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:31.584830 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" containerID="cri-o://28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c" gracePeriod=30 Nov 03 13:37:36.871377 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.871306 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:37:36.982359 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982331 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-istio-token\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-istio-token\") pod \"5864d40d-7211-41bd-990d-45d0e909224b\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " Nov 03 13:37:36.982539 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982370 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-envoy\") pod \"5864d40d-7211-41bd-990d-45d0e909224b\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " Nov 03 13:37:36.982539 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982403 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5864d40d-7211-41bd-990d-45d0e909224b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5864d40d-7211-41bd-990d-45d0e909224b-istiod-ca-cert\") pod \"5864d40d-7211-41bd-990d-45d0e909224b\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " Nov 03 13:37:36.982659 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982562 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-kube-api-access-crm88\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-kube-api-access-crm88\") pod \"5864d40d-7211-41bd-990d-45d0e909224b\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " Nov 03 13:37:36.982718 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982664 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-data\") pod \"5864d40d-7211-41bd-990d-45d0e909224b\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " Nov 03 13:37:36.982718 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982700 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-certs\") pod \"5864d40d-7211-41bd-990d-45d0e909224b\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " Nov 03 13:37:36.982833 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982790 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-socket\") pod \"5864d40d-7211-41bd-990d-45d0e909224b\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " Nov 03 13:37:36.982833 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982803 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5864d40d-7211-41bd-990d-45d0e909224b-istiod-ca-cert" pod "5864d40d-7211-41bd-990d-45d0e909224b" (UID: "5864d40d-7211-41bd-990d-45d0e909224b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:36.982906 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982836 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-credential-socket\") pod \"5864d40d-7211-41bd-990d-45d0e909224b\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " Nov 03 13:37:36.982906 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982889 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5864d40d-7211-41bd-990d-45d0e909224b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5864d40d-7211-41bd-990d-45d0e909224b-istio-podinfo\") pod \"5864d40d-7211-41bd-990d-45d0e909224b\" (UID: \"5864d40d-7211-41bd-990d-45d0e909224b\") " Nov 03 13:37:36.983014 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982973 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-data" pod "5864d40d-7211-41bd-990d-45d0e909224b" (UID: "5864d40d-7211-41bd-990d-45d0e909224b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.983014 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.982987 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-certs" pod "5864d40d-7211-41bd-990d-45d0e909224b" (UID: "5864d40d-7211-41bd-990d-45d0e909224b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.983233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.983193 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-credential-socket" pod "5864d40d-7211-41bd-990d-45d0e909224b" (UID: "5864d40d-7211-41bd-990d-45d0e909224b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.983785 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.983565 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.983944 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.983926 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5864d40d-7211-41bd-990d-45d0e909224b-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.984063 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.984040 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.984163 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.984065 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.984248 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.984228 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-socket" pod "5864d40d-7211-41bd-990d-45d0e909224b" (UID: "5864d40d-7211-41bd-990d-45d0e909224b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.985308 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.985270 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-envoy" pod "5864d40d-7211-41bd-990d-45d0e909224b" (UID: "5864d40d-7211-41bd-990d-45d0e909224b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.985418 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.985340 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5864d40d-7211-41bd-990d-45d0e909224b-istio-podinfo" pod "5864d40d-7211-41bd-990d-45d0e909224b" (UID: "5864d40d-7211-41bd-990d-45d0e909224b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:36.985652 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.985624 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-kube-api-access-crm88" pod "5864d40d-7211-41bd-990d-45d0e909224b" (UID: "5864d40d-7211-41bd-990d-45d0e909224b"). InnerVolumeSpecName "kube-api-access-crm88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:36.985756 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:36.985702 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-istio-token" pod "5864d40d-7211-41bd-990d-45d0e909224b" (UID: "5864d40d-7211-41bd-990d-45d0e909224b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:37.084890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.084845 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.084890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.084885 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.084890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.084896 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crm88\" (UniqueName: \"kubernetes.io/projected/5864d40d-7211-41bd-990d-45d0e909224b-kube-api-access-crm88\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.084890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.084907 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5864d40d-7211-41bd-990d-45d0e909224b-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.085163 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.084917 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5864d40d-7211-41bd-990d-45d0e909224b-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.400686 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.400652 4541 generic.go:356] "Generic (PLEG): container finished" podID="5864d40d-7211-41bd-990d-45d0e909224b" containerID="28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c" exitCode=0 Nov 03 13:37:37.400894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.400701 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" Nov 03 13:37:37.400894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.400727 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" event={"ID":"5864d40d-7211-41bd-990d-45d0e909224b","Type":"ContainerDied","Data":"28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c"} Nov 03 13:37:37.400894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.400786 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29" event={"ID":"5864d40d-7211-41bd-990d-45d0e909224b","Type":"ContainerDied","Data":"febe7f86c0bfc9bb522739c6c3423437b54ee5eff773ac8facb57b2d2e039ac9"} Nov 03 13:37:37.400894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.400815 4541 scope.go:122] "RemoveContainer" containerID="28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c" Nov 03 13:37:37.411912 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.411827 4541 scope.go:122] "RemoveContainer" containerID="28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c" Nov 03 13:37:37.412084 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:37:37.412062 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c\": container with ID starting with 28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c not found: ID does not exist" containerID="28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c" Nov 03 13:37:37.412129 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.412091 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c"} err="failed to get container status \"28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c\": rpc error: code = NotFound desc = could not find container \"28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c\": container with ID starting with 28fe076eff03af5eb7fa34a350515e881c0d76ac13144cf8f2347475a2df8a0c not found: ID does not exist" Nov 03 13:37:37.557613 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.557580 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29"] Nov 03 13:37:37.604457 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:37.604422 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6bg29"] Nov 03 13:37:38.868710 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:37:38.868674 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5864d40d-7211-41bd-990d-45d0e909224b" path="/var/lib/kubelet/pods/5864d40d-7211-41bd-990d-45d0e909224b/volumes" Nov 03 13:37:46.890757 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:37:46.890719 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:38:16.889735 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:38:16.889655 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:38:46.889842 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:38:46.889802 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:39:16.891185 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:39:16.891149 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:39:21.631646 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:21.631605 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk"] Nov 03 13:39:21.632131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:21.631847 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" containerID="cri-o://8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92" gracePeriod=30 Nov 03 13:39:22.514660 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:22.514621 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:39:22.514982 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:22.514696 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:39:26.897424 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.897396 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:39:26.974147 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974117 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-socket\") pod \"0b671863-3595-4ae5-a3fa-b8362ec220fa\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " Nov 03 13:39:26.974309 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974233 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-token\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-token\") pod \"0b671863-3595-4ae5-a3fa-b8362ec220fa\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " Nov 03 13:39:26.974309 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974279 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-certs\") pod \"0b671863-3595-4ae5-a3fa-b8362ec220fa\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " Nov 03 13:39:26.974309 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974300 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b671863-3595-4ae5-a3fa-b8362ec220fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b671863-3595-4ae5-a3fa-b8362ec220fa-istiod-ca-cert\") pod \"0b671863-3595-4ae5-a3fa-b8362ec220fa\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " Nov 03 13:39:26.974481 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974338 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-podinfo\") pod \"0b671863-3595-4ae5-a3fa-b8362ec220fa\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " Nov 03 13:39:26.974481 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974358 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-envoy\") pod \"0b671863-3595-4ae5-a3fa-b8362ec220fa\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " Nov 03 13:39:26.974481 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974383 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-credential-socket\") pod \"0b671863-3595-4ae5-a3fa-b8362ec220fa\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " Nov 03 13:39:26.974481 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974403 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-socket" pod "0b671863-3595-4ae5-a3fa-b8362ec220fa" (UID: "0b671863-3595-4ae5-a3fa-b8362ec220fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:26.974481 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974456 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-kube-api-access-w8vfq\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-kube-api-access-w8vfq\") pod \"0b671863-3595-4ae5-a3fa-b8362ec220fa\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " Nov 03 13:39:26.974740 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974541 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-data\") pod \"0b671863-3595-4ae5-a3fa-b8362ec220fa\" (UID: \"0b671863-3595-4ae5-a3fa-b8362ec220fa\") " Nov 03 13:39:26.974740 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974569 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-certs" pod "0b671863-3595-4ae5-a3fa-b8362ec220fa" (UID: "0b671863-3595-4ae5-a3fa-b8362ec220fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:26.974740 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974682 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-credential-socket" pod "0b671863-3595-4ae5-a3fa-b8362ec220fa" (UID: "0b671863-3595-4ae5-a3fa-b8362ec220fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:26.974955 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974818 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b671863-3595-4ae5-a3fa-b8362ec220fa-istiod-ca-cert" pod "0b671863-3595-4ae5-a3fa-b8362ec220fa" (UID: "0b671863-3595-4ae5-a3fa-b8362ec220fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:26.974994 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.974948 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-data" pod "0b671863-3595-4ae5-a3fa-b8362ec220fa" (UID: "0b671863-3595-4ae5-a3fa-b8362ec220fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:26.975265 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.975245 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:26.975316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.975272 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b671863-3595-4ae5-a3fa-b8362ec220fa-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:26.975316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.975288 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:26.975316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.975304 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:26.975410 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.975320 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:26.976970 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.976946 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-kube-api-access-w8vfq" pod "0b671863-3595-4ae5-a3fa-b8362ec220fa" (UID: "0b671863-3595-4ae5-a3fa-b8362ec220fa"). InnerVolumeSpecName "kube-api-access-w8vfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:26.977087 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.976981 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-token" pod "0b671863-3595-4ae5-a3fa-b8362ec220fa" (UID: "0b671863-3595-4ae5-a3fa-b8362ec220fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:26.977087 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.977038 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-podinfo" pod "0b671863-3595-4ae5-a3fa-b8362ec220fa" (UID: "0b671863-3595-4ae5-a3fa-b8362ec220fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:39:26.977087 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:26.977043 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-envoy" pod "0b671863-3595-4ae5-a3fa-b8362ec220fa" (UID: "0b671863-3595-4ae5-a3fa-b8362ec220fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:27.076758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.076717 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:27.076758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.076752 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:27.076758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.076782 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b671863-3595-4ae5-a3fa-b8362ec220fa-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:27.077145 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.076796 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8vfq\" (UniqueName: \"kubernetes.io/projected/0b671863-3595-4ae5-a3fa-b8362ec220fa-kube-api-access-w8vfq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:27.083399 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.083370 4541 generic.go:356] "Generic (PLEG): container finished" podID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerID="8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92" exitCode=0 Nov 03 13:39:27.083522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.083425 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" Nov 03 13:39:27.083522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.083445 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" event={"ID":"0b671863-3595-4ae5-a3fa-b8362ec220fa","Type":"ContainerDied","Data":"8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92"} Nov 03 13:39:27.083522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.083497 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk" event={"ID":"0b671863-3595-4ae5-a3fa-b8362ec220fa","Type":"ContainerDied","Data":"c39202922967f6d5f4a8e593b471a0477016326c0d8edec8cfb5b8e1efeea820"} Nov 03 13:39:27.083522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.083510 4541 scope.go:122] "RemoveContainer" containerID="8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92" Nov 03 13:39:27.094109 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.094085 4541 scope.go:122] "RemoveContainer" containerID="8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92" Nov 03 13:39:27.094354 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:39:27.094335 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92\": container with ID starting with 8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92 not found: ID does not exist" containerID="8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92" Nov 03 13:39:27.094402 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.094363 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92"} err="failed to get container status \"8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92\": rpc error: code = NotFound desc = could not find container \"8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92\": container with ID starting with 8122a41be4a84a8367686f4dbb787ed55946d298f571f08b2a5682c7be356c92 not found: ID does not exist" Nov 03 13:39:27.251800 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.251707 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk"] Nov 03 13:39:27.257750 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:27.257723 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8hcfk"] Nov 03 13:39:28.868377 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:28.868347 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" path="/var/lib/kubelet/pods/0b671863-3595-4ae5-a3fa-b8362ec220fa/volumes" Nov 03 13:39:29.120689 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:29.120608 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:39:29.122032 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:39:29.122005 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:39:46.890753 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:39:46.890713 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:40:16.890068 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:40:16.890028 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:40:46.890217 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:40:46.890171 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:41:16.889715 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:41:16.889627 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:41:33.388217 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:33.388178 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt"] Nov 03 13:41:33.388808 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:33.388410 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" containerID="cri-o://d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f" gracePeriod=30 Nov 03 13:41:35.285488 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:35.285449 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:41:35.285924 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:35.285526 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:41:37.731649 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:37.731612 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc"] Nov 03 13:41:37.732193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:37.731853 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" containerID="cri-o://60458ff7d58b30c51803cc59ad0082bd4c3f53efc418348f14790362044cc20b" gracePeriod=30 Nov 03 13:41:38.640873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.640846 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:41:38.738212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738124 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-envoy\") pod \"341f6a95-e931-4864-9caf-975236c1d7f9\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " Nov 03 13:41:38.738212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738165 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-kube-api-access-dj95c\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-kube-api-access-dj95c\") pod \"341f6a95-e931-4864-9caf-975236c1d7f9\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " Nov 03 13:41:38.738212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738212 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-socket\") pod \"341f6a95-e931-4864-9caf-975236c1d7f9\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738270 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-credential-socket\") pod \"341f6a95-e931-4864-9caf-975236c1d7f9\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738471 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-certs\") pod \"341f6a95-e931-4864-9caf-975236c1d7f9\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738552 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-credential-socket" pod "341f6a95-e931-4864-9caf-975236c1d7f9" (UID: "341f6a95-e931-4864-9caf-975236c1d7f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738558 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-data\") pod \"341f6a95-e931-4864-9caf-975236c1d7f9\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738568 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-socket" pod "341f6a95-e931-4864-9caf-975236c1d7f9" (UID: "341f6a95-e931-4864-9caf-975236c1d7f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738655 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341f6a95-e931-4864-9caf-975236c1d7f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341f6a95-e931-4864-9caf-975236c1d7f9-istiod-ca-cert\") pod \"341f6a95-e931-4864-9caf-975236c1d7f9\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738699 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-certs" pod "341f6a95-e931-4864-9caf-975236c1d7f9" (UID: "341f6a95-e931-4864-9caf-975236c1d7f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738715 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-istio-token\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-istio-token\") pod \"341f6a95-e931-4864-9caf-975236c1d7f9\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738732 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-data" pod "341f6a95-e931-4864-9caf-975236c1d7f9" (UID: "341f6a95-e931-4864-9caf-975236c1d7f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:38.738827 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.738757 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/341f6a95-e931-4864-9caf-975236c1d7f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341f6a95-e931-4864-9caf-975236c1d7f9-istio-podinfo\") pod \"341f6a95-e931-4864-9caf-975236c1d7f9\" (UID: \"341f6a95-e931-4864-9caf-975236c1d7f9\") " Nov 03 13:41:38.739319 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.739058 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341f6a95-e931-4864-9caf-975236c1d7f9-istiod-ca-cert" pod "341f6a95-e931-4864-9caf-975236c1d7f9" (UID: "341f6a95-e931-4864-9caf-975236c1d7f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:38.739375 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.739345 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:38.739375 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.739366 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/341f6a95-e931-4864-9caf-975236c1d7f9-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:38.739466 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.739384 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:38.739466 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.739398 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:38.739466 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.739412 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:38.740541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.740518 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-kube-api-access-dj95c" pod "341f6a95-e931-4864-9caf-975236c1d7f9" (UID: "341f6a95-e931-4864-9caf-975236c1d7f9"). InnerVolumeSpecName "kube-api-access-dj95c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:38.740644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.740568 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-envoy" pod "341f6a95-e931-4864-9caf-975236c1d7f9" (UID: "341f6a95-e931-4864-9caf-975236c1d7f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:38.741141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.741123 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/341f6a95-e931-4864-9caf-975236c1d7f9-istio-podinfo" pod "341f6a95-e931-4864-9caf-975236c1d7f9" (UID: "341f6a95-e931-4864-9caf-975236c1d7f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:41:38.741208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.741158 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-istio-token" pod "341f6a95-e931-4864-9caf-975236c1d7f9" (UID: "341f6a95-e931-4864-9caf-975236c1d7f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:38.839331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.839295 4541 generic.go:356] "Generic (PLEG): container finished" podID="341f6a95-e931-4864-9caf-975236c1d7f9" containerID="d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f" exitCode=0 Nov 03 13:41:38.839507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.839356 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" Nov 03 13:41:38.839507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.839382 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" event={"ID":"341f6a95-e931-4864-9caf-975236c1d7f9","Type":"ContainerDied","Data":"d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f"} Nov 03 13:41:38.839507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.839475 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt" event={"ID":"341f6a95-e931-4864-9caf-975236c1d7f9","Type":"ContainerDied","Data":"b4b0b9b313f04192e48bb4a053f31fdf49d7b6707b4d30d268cbe009174c6e65"} Nov 03 13:41:38.839618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.839506 4541 scope.go:122] "RemoveContainer" containerID="d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f" Nov 03 13:41:38.840439 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.840413 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:38.840534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.840445 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/341f6a95-e931-4864-9caf-975236c1d7f9-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:38.840534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.840460 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/341f6a95-e931-4864-9caf-975236c1d7f9-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:38.840534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.840474 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj95c\" (UniqueName: \"kubernetes.io/projected/341f6a95-e931-4864-9caf-975236c1d7f9-kube-api-access-dj95c\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:38.850926 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.850910 4541 scope.go:122] "RemoveContainer" containerID="d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f" Nov 03 13:41:38.851198 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:41:38.851182 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f\": container with ID starting with d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f not found: ID does not exist" containerID="d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f" Nov 03 13:41:38.851244 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.851205 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f"} err="failed to get container status \"d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f\": rpc error: code = NotFound desc = could not find container \"d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f\": container with ID starting with d826bbbbad9a21572213e64fe69ec3547b88c9b6a3ac425fee68f26a238a748f not found: ID does not exist" Nov 03 13:41:38.999651 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:38.999559 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt"] Nov 03 13:41:39.006001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:39.005972 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-vq7vt"] Nov 03 13:41:40.868356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:40.868322 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" path="/var/lib/kubelet/pods/341f6a95-e931-4864-9caf-975236c1d7f9/volumes" Nov 03 13:41:42.528187 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:42.528152 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5"] Nov 03 13:41:42.528684 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:42.528407 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" containerID="cri-o://28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09" gracePeriod=30 Nov 03 13:41:42.873072 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:42.873037 4541 generic.go:356] "Generic (PLEG): container finished" podID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerID="60458ff7d58b30c51803cc59ad0082bd4c3f53efc418348f14790362044cc20b" exitCode=0 Nov 03 13:41:42.873072 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:42.873074 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" event={"ID":"91603f00-f1dd-4367-9d41-27c3dcb09ac7","Type":"ContainerDied","Data":"60458ff7d58b30c51803cc59ad0082bd4c3f53efc418348f14790362044cc20b"} Nov 03 13:41:43.002661 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.002631 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:41:43.088289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088248 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-socket\") pod \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " Nov 03 13:41:43.088469 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088404 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-kube-api-access-t4srt\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-kube-api-access-t4srt\") pod \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " Nov 03 13:41:43.088469 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088463 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-data\") pod \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " Nov 03 13:41:43.088568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088485 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-socket" pod "91603f00-f1dd-4367-9d41-27c3dcb09ac7" (UID: "91603f00-f1dd-4367-9d41-27c3dcb09ac7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:43.088568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088503 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-podinfo\") pod \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " Nov 03 13:41:43.088568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088529 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-token\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-token\") pod \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " Nov 03 13:41:43.088568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088545 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-credential-socket\") pod \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " Nov 03 13:41:43.088735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088574 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istiod-ca-cert\") pod \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " Nov 03 13:41:43.088735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088666 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-certs\") pod \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " Nov 03 13:41:43.088735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088694 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-envoy\") pod \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\" (UID: \"91603f00-f1dd-4367-9d41-27c3dcb09ac7\") " Nov 03 13:41:43.088735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088704 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-data" pod "91603f00-f1dd-4367-9d41-27c3dcb09ac7" (UID: "91603f00-f1dd-4367-9d41-27c3dcb09ac7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:43.088981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088887 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-credential-socket" pod "91603f00-f1dd-4367-9d41-27c3dcb09ac7" (UID: "91603f00-f1dd-4367-9d41-27c3dcb09ac7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:43.088981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.088967 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-certs" pod "91603f00-f1dd-4367-9d41-27c3dcb09ac7" (UID: "91603f00-f1dd-4367-9d41-27c3dcb09ac7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:43.089138 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.089103 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istiod-ca-cert" pod "91603f00-f1dd-4367-9d41-27c3dcb09ac7" (UID: "91603f00-f1dd-4367-9d41-27c3dcb09ac7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:43.089272 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.089259 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:43.089336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.089279 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:43.089336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.089294 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:43.089336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.089307 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:43.089336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.089321 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:43.090888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.090864 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-token" pod "91603f00-f1dd-4367-9d41-27c3dcb09ac7" (UID: "91603f00-f1dd-4367-9d41-27c3dcb09ac7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:43.090888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.090879 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-kube-api-access-t4srt" pod "91603f00-f1dd-4367-9d41-27c3dcb09ac7" (UID: "91603f00-f1dd-4367-9d41-27c3dcb09ac7"). InnerVolumeSpecName "kube-api-access-t4srt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:43.091049 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.091012 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-podinfo" pod "91603f00-f1dd-4367-9d41-27c3dcb09ac7" (UID: "91603f00-f1dd-4367-9d41-27c3dcb09ac7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:41:43.091142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.091126 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-envoy" pod "91603f00-f1dd-4367-9d41-27c3dcb09ac7" (UID: "91603f00-f1dd-4367-9d41-27c3dcb09ac7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:43.190196 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.190161 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:43.190196 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.190190 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:43.190196 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.190200 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91603f00-f1dd-4367-9d41-27c3dcb09ac7-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:43.190436 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.190216 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4srt\" (UniqueName: \"kubernetes.io/projected/91603f00-f1dd-4367-9d41-27c3dcb09ac7-kube-api-access-t4srt\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:43.879893 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.879861 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" Nov 03 13:41:43.879893 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.879872 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc" event={"ID":"91603f00-f1dd-4367-9d41-27c3dcb09ac7","Type":"ContainerDied","Data":"e3154e43fa5cc1a66f1e5c0092a036f19f07684b51a6c70923e44ac18e7ace9e"} Nov 03 13:41:43.880399 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:43.879930 4541 scope.go:122] "RemoveContainer" containerID="60458ff7d58b30c51803cc59ad0082bd4c3f53efc418348f14790362044cc20b" Nov 03 13:41:44.042212 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:44.042177 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc"] Nov 03 13:41:44.089662 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:44.089624 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-cwzrc"] Nov 03 13:41:44.868339 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:44.868305 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" path="/var/lib/kubelet/pods/91603f00-f1dd-4367-9d41-27c3dcb09ac7/volumes" Nov 03 13:41:46.891420 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:41:46.891372 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:41:47.780303 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.780266 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:41:47.838450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838373 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-certs\") pod \"ce2441ba-5c60-47a7-bad3-749229ec5296\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " Nov 03 13:41:47.838450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838417 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-socket\") pod \"ce2441ba-5c60-47a7-bad3-749229ec5296\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " Nov 03 13:41:47.838450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838446 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2441ba-5c60-47a7-bad3-749229ec5296-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2441ba-5c60-47a7-bad3-749229ec5296-istiod-ca-cert\") pod \"ce2441ba-5c60-47a7-bad3-749229ec5296\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " Nov 03 13:41:47.838735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838494 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-data\") pod \"ce2441ba-5c60-47a7-bad3-749229ec5296\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " Nov 03 13:41:47.838735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838518 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-podinfo\") pod \"ce2441ba-5c60-47a7-bad3-749229ec5296\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " Nov 03 13:41:47.838735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838537 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-credential-socket\") pod \"ce2441ba-5c60-47a7-bad3-749229ec5296\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " Nov 03 13:41:47.838735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838589 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-envoy\") pod \"ce2441ba-5c60-47a7-bad3-749229ec5296\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " Nov 03 13:41:47.838735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838663 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-token\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-token\") pod \"ce2441ba-5c60-47a7-bad3-749229ec5296\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " Nov 03 13:41:47.838735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838662 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-certs" pod "ce2441ba-5c60-47a7-bad3-749229ec5296" (UID: "ce2441ba-5c60-47a7-bad3-749229ec5296"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:47.838735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838691 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-kube-api-access-dvh6d\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-kube-api-access-dvh6d\") pod \"ce2441ba-5c60-47a7-bad3-749229ec5296\" (UID: \"ce2441ba-5c60-47a7-bad3-749229ec5296\") " Nov 03 13:41:47.839123 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838800 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-socket" pod "ce2441ba-5c60-47a7-bad3-749229ec5296" (UID: "ce2441ba-5c60-47a7-bad3-749229ec5296"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:47.839123 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838861 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-data" pod "ce2441ba-5c60-47a7-bad3-749229ec5296" (UID: "ce2441ba-5c60-47a7-bad3-749229ec5296"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:47.839123 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838976 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-credential-socket" pod "ce2441ba-5c60-47a7-bad3-749229ec5296" (UID: "ce2441ba-5c60-47a7-bad3-749229ec5296"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:47.839123 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.838997 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2441ba-5c60-47a7-bad3-749229ec5296-istiod-ca-cert" pod "ce2441ba-5c60-47a7-bad3-749229ec5296" (UID: "ce2441ba-5c60-47a7-bad3-749229ec5296"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:47.839324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.839238 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:47.839324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.839262 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:47.839324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.839277 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce2441ba-5c60-47a7-bad3-749229ec5296-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:47.839324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.839290 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:47.839324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.839304 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:47.841033 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.841004 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-token" pod "ce2441ba-5c60-47a7-bad3-749229ec5296" (UID: "ce2441ba-5c60-47a7-bad3-749229ec5296"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:47.841143 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.841051 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-podinfo" pod "ce2441ba-5c60-47a7-bad3-749229ec5296" (UID: "ce2441ba-5c60-47a7-bad3-749229ec5296"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:41:47.841143 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.841057 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-kube-api-access-dvh6d" pod "ce2441ba-5c60-47a7-bad3-749229ec5296" (UID: "ce2441ba-5c60-47a7-bad3-749229ec5296"). InnerVolumeSpecName "kube-api-access-dvh6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:47.841143 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.841067 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-envoy" pod "ce2441ba-5c60-47a7-bad3-749229ec5296" (UID: "ce2441ba-5c60-47a7-bad3-749229ec5296"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:47.903734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.903701 4541 generic.go:356] "Generic (PLEG): container finished" podID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerID="28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09" exitCode=0 Nov 03 13:41:47.904157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.903750 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" Nov 03 13:41:47.904157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.903808 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" event={"ID":"ce2441ba-5c60-47a7-bad3-749229ec5296","Type":"ContainerDied","Data":"28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09"} Nov 03 13:41:47.904157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.903861 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5" event={"ID":"ce2441ba-5c60-47a7-bad3-749229ec5296","Type":"ContainerDied","Data":"aa7ec9b4cd541452fa81c04e4d97b3a382d621259a2af56fb47e56ee4b6976e5"} Nov 03 13:41:47.904157 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.903882 4541 scope.go:122] "RemoveContainer" containerID="28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09" Nov 03 13:41:47.913549 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.913532 4541 scope.go:122] "RemoveContainer" containerID="28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09" Nov 03 13:41:47.913839 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:41:47.913814 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09\": container with ID starting with 28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09 not found: ID does not exist" containerID="28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09" Nov 03 13:41:47.913902 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.913847 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09"} err="failed to get container status \"28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09\": rpc error: code = NotFound desc = could not find container \"28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09\": container with ID starting with 28f73fa6cceaa2f1d5bc1fd9bf6ef93dddf4e11e355b00f211c262fb9e2dbb09 not found: ID does not exist" Nov 03 13:41:47.941164 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.941124 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:47.941164 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.941164 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvh6d\" (UniqueName: \"kubernetes.io/projected/ce2441ba-5c60-47a7-bad3-749229ec5296-kube-api-access-dvh6d\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:47.941386 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.941182 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:47.941386 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:47.941198 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce2441ba-5c60-47a7-bad3-749229ec5296-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:48.051581 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:48.051543 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5"] Nov 03 13:41:48.057872 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:48.057843 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-26hn5"] Nov 03 13:41:48.869088 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:41:48.869051 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" path="/var/lib/kubelet/pods/ce2441ba-5c60-47a7-bad3-749229ec5296/volumes" Nov 03 13:42:00.347865 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.347829 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586"] Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348308 4541 cpu_manager.go:414] "containerMap: removing container" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348320 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348332 4541 cpu_manager.go:414] "containerMap: removing container" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348338 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348347 4541 cpu_manager.go:414] "containerMap: removing container" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348354 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348364 4541 cpu_manager.go:414] "containerMap: removing container" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348369 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348379 4541 cpu_manager.go:414] "containerMap: removing container" podUID="347b5f6e-0def-408a-9739-fceb7e2d647b" containerName="maas-api" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348384 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347b5f6e-0def-408a-9739-fceb7e2d647b" containerName="maas-api" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348395 4541 cpu_manager.go:414] "containerMap: removing container" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerName="istio-proxy" Nov 03 13:42:00.348394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348400 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348409 4541 cpu_manager.go:414] "containerMap: removing container" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348413 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348427 4541 cpu_manager.go:414] "containerMap: removing container" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348431 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348437 4541 cpu_manager.go:414] "containerMap: removing container" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348441 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348449 4541 cpu_manager.go:414] "containerMap: removing container" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348453 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348459 4541 cpu_manager.go:414] "containerMap: removing container" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348464 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348471 4541 cpu_manager.go:414] "containerMap: removing container" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348475 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348481 4541 cpu_manager.go:414] "containerMap: removing container" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348485 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348493 4541 cpu_manager.go:414] "containerMap: removing container" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348497 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348618 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f98690-9c1c-462c-b1d2-a9d4a5ef11b8" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348634 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="112a1add-c290-4f08-b1c5-48118ce06a2b" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348644 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b671863-3595-4ae5-a3fa-b8362ec220fa" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348651 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="341f6a95-e931-4864-9caf-975236c1d7f9" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348657 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb2e35f1-95b8-4488-95f4-b461b2666007" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348667 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="37545439-ddd2-40d6-bf2f-a2b8c926ad5e" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348672 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="86bbf92d-72fe-49d6-9e1b-6265bdf1b809" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348678 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef1e1f4-063c-4daa-8a31-f40d7bd561d5" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348685 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="5864d40d-7211-41bd-990d-45d0e909224b" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348692 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="347b5f6e-0def-408a-9739-fceb7e2d647b" containerName="maas-api" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348702 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c33209-ae02-49e1-877d-4fbf55a18c4c" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348710 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2441ba-5c60-47a7-bad3-749229ec5296" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348721 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="91603f00-f1dd-4367-9d41-27c3dcb09ac7" containerName="istio-proxy" Nov 03 13:42:00.348930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.348728 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="e017540c-faef-4bae-ab8e-ca161785ad52" containerName="istio-proxy" Nov 03 13:42:00.352051 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.352033 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:00.463686 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.463652 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-84e22c\"" Nov 03 13:42:00.463997 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.463977 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-84e22c-dockercfg-pgcj4\"" Nov 03 13:42:00.473541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.473504 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586"] Nov 03 13:42:00.484241 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.484201 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98100ef-ccee-4d2d-af02-9f2d8cd87289-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586\" (UID: \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:00.484512 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.484485 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqvz7\" (UniqueName: \"kubernetes.io/projected/b98100ef-ccee-4d2d-af02-9f2d8cd87289-kube-api-access-kqvz7\") pod \"payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586\" (UID: \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:00.508328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.508291 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj"] Nov 03 13:42:00.512352 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.512323 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:00.585667 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.585627 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98100ef-ccee-4d2d-af02-9f2d8cd87289-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586\" (UID: \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:00.585879 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.585687 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2vv\" (UniqueName: \"kubernetes.io/projected/55438f58-edb0-41ba-bc20-0abcd94dfc21-kube-api-access-nh2vv\") pod \"payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj\" (UID: \"55438f58-edb0-41ba-bc20-0abcd94dfc21\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:00.585879 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.585728 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqvz7\" (UniqueName: \"kubernetes.io/projected/b98100ef-ccee-4d2d-af02-9f2d8cd87289-kube-api-access-kqvz7\") pod \"payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586\" (UID: \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:00.585991 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.585941 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55438f58-edb0-41ba-bc20-0abcd94dfc21-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj\" (UID: \"55438f58-edb0-41ba-bc20-0abcd94dfc21\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:00.586498 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.586475 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98100ef-ccee-4d2d-af02-9f2d8cd87289-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586\" (UID: \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:00.587964 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.587938 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj"] Nov 03 13:42:00.646060 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.645983 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqvz7\" (UniqueName: \"kubernetes.io/projected/b98100ef-ccee-4d2d-af02-9f2d8cd87289-kube-api-access-kqvz7\") pod \"payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586\" (UID: \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:00.666124 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.666090 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:00.687370 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.687341 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55438f58-edb0-41ba-bc20-0abcd94dfc21-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj\" (UID: \"55438f58-edb0-41ba-bc20-0abcd94dfc21\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:00.687529 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.687455 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2vv\" (UniqueName: \"kubernetes.io/projected/55438f58-edb0-41ba-bc20-0abcd94dfc21-kube-api-access-nh2vv\") pod \"payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj\" (UID: \"55438f58-edb0-41ba-bc20-0abcd94dfc21\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:00.688113 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.688089 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55438f58-edb0-41ba-bc20-0abcd94dfc21-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj\" (UID: \"55438f58-edb0-41ba-bc20-0abcd94dfc21\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:00.743308 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.743265 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2vv\" (UniqueName: \"kubernetes.io/projected/55438f58-edb0-41ba-bc20-0abcd94dfc21-kube-api-access-nh2vv\") pod \"payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj\" (UID: \"55438f58-edb0-41ba-bc20-0abcd94dfc21\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:00.826792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:00.826742 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:01.045801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:01.044927 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586"] Nov 03 13:42:01.047809 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:42:01.047030 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98100ef_ccee_4d2d_af02_9f2d8cd87289.slice/crio-93c8423a109045d6fb28a259eec205300dc190851ce0d5c6d05518297702f415 WatchSource:0}: Error finding container 93c8423a109045d6fb28a259eec205300dc190851ce0d5c6d05518297702f415: Status 404 returned error can't find the container with id 93c8423a109045d6fb28a259eec205300dc190851ce0d5c6d05518297702f415 Nov 03 13:42:01.206154 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:01.206117 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj"] Nov 03 13:42:01.207584 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:42:01.207557 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55438f58_edb0_41ba_bc20_0abcd94dfc21.slice/crio-483185632a73ecfe5fbfd71392e416317ffc7fa1713bd0ac6565b1e53482666c WatchSource:0}: Error finding container 483185632a73ecfe5fbfd71392e416317ffc7fa1713bd0ac6565b1e53482666c: Status 404 returned error can't find the container with id 483185632a73ecfe5fbfd71392e416317ffc7fa1713bd0ac6565b1e53482666c Nov 03 13:42:01.982918 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:01.982880 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" event={"ID":"55438f58-edb0-41ba-bc20-0abcd94dfc21","Type":"ContainerStarted","Data":"b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45"} Nov 03 13:42:01.983427 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:01.982938 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" event={"ID":"55438f58-edb0-41ba-bc20-0abcd94dfc21","Type":"ContainerStarted","Data":"483185632a73ecfe5fbfd71392e416317ffc7fa1713bd0ac6565b1e53482666c"} Nov 03 13:42:01.983427 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:01.982965 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:01.984306 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:01.984278 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" event={"ID":"b98100ef-ccee-4d2d-af02-9f2d8cd87289","Type":"ContainerStarted","Data":"e129479b9858b2bcc50b811613444fdd0847feb183099090b96d75e87d86d80e"} Nov 03 13:42:01.984378 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:01.984324 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" event={"ID":"b98100ef-ccee-4d2d-af02-9f2d8cd87289","Type":"ContainerStarted","Data":"93c8423a109045d6fb28a259eec205300dc190851ce0d5c6d05518297702f415"} Nov 03 13:42:01.984446 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:01.984373 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:02.099097 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:02.099042 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" podStartSLOduration=2.09902865 podStartE2EDuration="2.09902865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:02.096167502 +0000 UTC m=+2253.410139363" watchObservedRunningTime="2026-11-03 13:42:02.09902865 +0000 UTC m=+2253.413000510" Nov 03 13:42:02.214079 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:02.214022 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" podStartSLOduration=2.214006243 podStartE2EDuration="2.214006243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:02.213767118 +0000 UTC m=+2253.527738977" watchObservedRunningTime="2026-11-03 13:42:02.214006243 +0000 UTC m=+2253.527978103" Nov 03 13:42:03.521027 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:03.520992 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-qnx22_bf6b3624-855b-4ac2-88d4-cfa28b23a721/payload-processing/0.log" Nov 03 13:42:03.788111 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:03.788026 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-qnx22_bf6b3624-855b-4ac2-88d4-cfa28b23a721/payload-processing/0.log" Nov 03 13:42:12.996008 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:12.995976 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:42:12.996547 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:42:12.996058 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:42:16.901278 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:42:16.901230 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:42:46.889468 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:42:46.889432 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:43:16.895243 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:43:16.895205 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:43:46.889458 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:43:46.889420 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:44:16.889757 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:44:16.889673 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:44:25.717484 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:44:25.717448 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-qnx22_bf6b3624-855b-4ac2-88d4-cfa28b23a721/payload-processing/0.log" Nov 03 13:44:25.991145 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:44:25.991065 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-qnx22_bf6b3624-855b-4ac2-88d4-cfa28b23a721/payload-processing/0.log" Nov 03 13:44:29.157575 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:44:29.157545 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:44:29.159302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:44:29.159279 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:44:46.889414 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:44:46.889374 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:44:59.378862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:44:59.378824 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf"] Nov 03 13:44:59.379511 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:44:59.379046 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" containerID="cri-o://3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941" gracePeriod=30 Nov 03 13:45:00.288126 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.287999 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x"] Nov 03 13:45:00.293907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.293881 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn"] Nov 03 13:45:00.294065 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.293994 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:00.302098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.302062 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:00.355615 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.355581 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x"] Nov 03 13:45:00.395177 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.395149 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d9fbf2-bde0-4801-8e6c-3882553894f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn\" (UID: \"74d9fbf2-bde0-4801-8e6c-3882553894f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:00.395544 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.395205 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091370ad-07c4-41c9-99e5-e990c02eb67a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895225-b9x5x\" (UID: \"091370ad-07c4-41c9-99e5-e990c02eb67a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:00.395544 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.395226 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xxc\" (UniqueName: \"kubernetes.io/projected/091370ad-07c4-41c9-99e5-e990c02eb67a-kube-api-access-59xxc\") pod \"maas-api-key-cleanup-29895225-b9x5x\" (UID: \"091370ad-07c4-41c9-99e5-e990c02eb67a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:00.395544 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.395341 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2p28\" (UniqueName: \"kubernetes.io/projected/74d9fbf2-bde0-4801-8e6c-3882553894f5-kube-api-access-q2p28\") pod \"maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn\" (UID: \"74d9fbf2-bde0-4801-8e6c-3882553894f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:00.395864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.395850 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:45:00.398805 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.398753 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn"] Nov 03 13:45:00.496520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.496486 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d9fbf2-bde0-4801-8e6c-3882553894f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn\" (UID: \"74d9fbf2-bde0-4801-8e6c-3882553894f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:00.496706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.496549 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091370ad-07c4-41c9-99e5-e990c02eb67a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895225-b9x5x\" (UID: \"091370ad-07c4-41c9-99e5-e990c02eb67a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:00.496706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.496569 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59xxc\" (UniqueName: \"kubernetes.io/projected/091370ad-07c4-41c9-99e5-e990c02eb67a-kube-api-access-59xxc\") pod \"maas-api-key-cleanup-29895225-b9x5x\" (UID: \"091370ad-07c4-41c9-99e5-e990c02eb67a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:00.496706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.496610 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2p28\" (UniqueName: \"kubernetes.io/projected/74d9fbf2-bde0-4801-8e6c-3882553894f5-kube-api-access-q2p28\") pod \"maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn\" (UID: \"74d9fbf2-bde0-4801-8e6c-3882553894f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:00.497300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.497277 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091370ad-07c4-41c9-99e5-e990c02eb67a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895225-b9x5x\" (UID: \"091370ad-07c4-41c9-99e5-e990c02eb67a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:00.497363 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.497328 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d9fbf2-bde0-4801-8e6c-3882553894f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn\" (UID: \"74d9fbf2-bde0-4801-8e6c-3882553894f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:00.553185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.553123 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xxc\" (UniqueName: \"kubernetes.io/projected/091370ad-07c4-41c9-99e5-e990c02eb67a-kube-api-access-59xxc\") pod \"maas-api-key-cleanup-29895225-b9x5x\" (UID: \"091370ad-07c4-41c9-99e5-e990c02eb67a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:00.553322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.553297 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2p28\" (UniqueName: \"kubernetes.io/projected/74d9fbf2-bde0-4801-8e6c-3882553894f5-kube-api-access-q2p28\") pod \"maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn\" (UID: \"74d9fbf2-bde0-4801-8e6c-3882553894f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:00.628478 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.628444 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:00.638666 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:00.638642 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:01.023273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:01.023240 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x"] Nov 03 13:45:01.023442 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:45:01.023414 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091370ad_07c4_41c9_99e5_e990c02eb67a.slice/crio-8ad34e6789fad2850821c84c7b0d21facbc41f5ac037e1f91cb3d5fe71b5d3b7 WatchSource:0}: Error finding container 8ad34e6789fad2850821c84c7b0d21facbc41f5ac037e1f91cb3d5fe71b5d3b7: Status 404 returned error can't find the container with id 8ad34e6789fad2850821c84c7b0d21facbc41f5ac037e1f91cb3d5fe71b5d3b7 Nov 03 13:45:01.062234 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:01.062205 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn"] Nov 03 13:45:01.065000 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:45:01.064968 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d9fbf2_bde0_4801_8e6c_3882553894f5.slice/crio-94e8f4d91740d099069960a099a6fd2a414aee4a9264c8f7cc075d2895e74f66 WatchSource:0}: Error finding container 94e8f4d91740d099069960a099a6fd2a414aee4a9264c8f7cc075d2895e74f66: Status 404 returned error can't find the container with id 94e8f4d91740d099069960a099a6fd2a414aee4a9264c8f7cc075d2895e74f66 Nov 03 13:45:02.003717 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.003683 4541 generic.go:356] "Generic (PLEG): container finished" podID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerID="df40269116c047c0df86f3561d8befb4d9ca9941d137ecaf8745706620233c07" exitCode=60 Nov 03 13:45:02.004270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.003766 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" event={"ID":"74d9fbf2-bde0-4801-8e6c-3882553894f5","Type":"ContainerDied","Data":"df40269116c047c0df86f3561d8befb4d9ca9941d137ecaf8745706620233c07"} Nov 03 13:45:02.004270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.003857 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" event={"ID":"74d9fbf2-bde0-4801-8e6c-3882553894f5","Type":"ContainerStarted","Data":"94e8f4d91740d099069960a099a6fd2a414aee4a9264c8f7cc075d2895e74f66"} Nov 03 13:45:02.004270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.004164 4541 scope.go:122] "RemoveContainer" containerID="df40269116c047c0df86f3561d8befb4d9ca9941d137ecaf8745706620233c07" Nov 03 13:45:02.005585 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.005560 4541 generic.go:356] "Generic (PLEG): container finished" podID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerID="17cc214c805e402c6a5e0ab36fd579df057e436fab61506006e7bd990072249c" exitCode=60 Nov 03 13:45:02.005706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.005655 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" event={"ID":"091370ad-07c4-41c9-99e5-e990c02eb67a","Type":"ContainerDied","Data":"17cc214c805e402c6a5e0ab36fd579df057e436fab61506006e7bd990072249c"} Nov 03 13:45:02.005798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.005712 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" event={"ID":"091370ad-07c4-41c9-99e5-e990c02eb67a","Type":"ContainerStarted","Data":"8ad34e6789fad2850821c84c7b0d21facbc41f5ac037e1f91cb3d5fe71b5d3b7"} Nov 03 13:45:02.006014 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.005998 4541 scope.go:122] "RemoveContainer" containerID="17cc214c805e402c6a5e0ab36fd579df057e436fab61506006e7bd990072249c" Nov 03 13:45:02.674278 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.674228 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24"] Nov 03 13:45:02.674542 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.674502 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" podUID="cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" containerName="discovery" containerID="cri-o://25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb" gracePeriod=30 Nov 03 13:45:02.976867 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:02.976842 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:45:03.013060 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.013024 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" event={"ID":"74d9fbf2-bde0-4801-8e6c-3882553894f5","Type":"ContainerStarted","Data":"5757a09dd5313ddc1507fb1b27fa2113f3eabff5e4457552c40963a7e1da52e8"} Nov 03 13:45:03.014409 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.014380 4541 generic.go:356] "Generic (PLEG): container finished" podID="cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" containerID="25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb" exitCode=0 Nov 03 13:45:03.014544 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.014418 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" Nov 03 13:45:03.014544 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.014462 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" event={"ID":"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1","Type":"ContainerDied","Data":"25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb"} Nov 03 13:45:03.014544 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.014501 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24" event={"ID":"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1","Type":"ContainerDied","Data":"188fc2deb35be29fd2f4f376cc35006a2e51ebbef39f47535d440fdbb80ea884"} Nov 03 13:45:03.014544 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.014522 4541 scope.go:122] "RemoveContainer" containerID="25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb" Nov 03 13:45:03.016403 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.016382 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" event={"ID":"091370ad-07c4-41c9-99e5-e990c02eb67a","Type":"ContainerStarted","Data":"81651bee9b3dc2ebdc32aefbc6dc73e084c89efbbf8fc6a310a49214ee308f74"} Nov 03 13:45:03.024194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.024177 4541 scope.go:122] "RemoveContainer" containerID="25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb" Nov 03 13:45:03.024475 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:03.024454 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb\": container with ID starting with 25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb not found: ID does not exist" containerID="25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb" Nov 03 13:45:03.024549 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.024482 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb"} err="failed to get container status \"25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb\": rpc error: code = NotFound desc = could not find container \"25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb\": container with ID starting with 25535b4c01ada5cefdde8586f911cb58cb773d251cbf5a388c7f86f71de47bdb not found: ID does not exist" Nov 03 13:45:03.026814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.026793 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-ca-configmap\") pod \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " Nov 03 13:45:03.026897 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.026833 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-cacerts\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-cacerts\") pod \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " Nov 03 13:45:03.026897 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.026882 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-local-certs\") pod \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " Nov 03 13:45:03.026985 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.026969 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-kube-api-access-7k4dk\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-kube-api-access-7k4dk\") pod \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " Nov 03 13:45:03.027040 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.027013 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-kubeconfig\") pod \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " Nov 03 13:45:03.027090 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.027076 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-token\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-token\") pod \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " Nov 03 13:45:03.027146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.027100 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-dns-cert\") pod \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\" (UID: \"cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1\") " Nov 03 13:45:03.027261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.027236 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-ca-configmap" pod "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" (UID: "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:03.027567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.027541 4541 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-ca-configmap\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:03.029387 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.029357 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-dns-cert" pod "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" (UID: "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:03.029387 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.029377 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-kubeconfig" pod "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" (UID: "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:03.029618 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.029596 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-cacerts" pod "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" (UID: "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:03.029709 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.029692 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-kube-api-access-7k4dk" pod "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" (UID: "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1"). InnerVolumeSpecName "kube-api-access-7k4dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:03.029837 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.029815 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-local-certs" pod "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" (UID: "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:03.029928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.029912 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-token" pod "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" (UID: "cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:03.135862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.129687 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k4dk\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-kube-api-access-7k4dk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:03.135862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.129735 4541 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-kubeconfig\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:03.135862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.129756 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:03.135862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.129793 4541 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-istio-csr-dns-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:03.135862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.129810 4541 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-cacerts\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:03.135862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.129830 4541 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1-local-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:03.212254 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.212142 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" podStartSLOduration=3.212127956 podStartE2EDuration="3.212127956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:03.208961933 +0000 UTC m=+2434.522933806" watchObservedRunningTime="2026-11-03 13:45:03.212127956 +0000 UTC m=+2434.526099865" Nov 03 13:45:03.377570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.377515 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" podStartSLOduration=3.377500748 podStartE2EDuration="3.377500748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:03.37510968 +0000 UTC m=+2434.689081540" watchObservedRunningTime="2026-11-03 13:45:03.377500748 +0000 UTC m=+2434.691472652" Nov 03 13:45:03.487721 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.487636 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24"] Nov 03 13:45:03.492072 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:03.492043 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-xwx24"] Nov 03 13:45:04.023936 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.023900 4541 generic.go:356] "Generic (PLEG): container finished" podID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerID="5757a09dd5313ddc1507fb1b27fa2113f3eabff5e4457552c40963a7e1da52e8" exitCode=60 Nov 03 13:45:04.024418 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.023968 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" event={"ID":"74d9fbf2-bde0-4801-8e6c-3882553894f5","Type":"ContainerDied","Data":"5757a09dd5313ddc1507fb1b27fa2113f3eabff5e4457552c40963a7e1da52e8"} Nov 03 13:45:04.024418 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.024039 4541 scope.go:122] "RemoveContainer" containerID="df40269116c047c0df86f3561d8befb4d9ca9941d137ecaf8745706620233c07" Nov 03 13:45:04.024418 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.024301 4541 scope.go:122] "RemoveContainer" containerID="5757a09dd5313ddc1507fb1b27fa2113f3eabff5e4457552c40963a7e1da52e8" Nov 03 13:45:04.024642 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:04.024615 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn_odh-ai-gateway-infra(74d9fbf2-bde0-4801-8e6c-3882553894f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" Nov 03 13:45:04.026951 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.026890 4541 generic.go:356] "Generic (PLEG): container finished" podID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerID="81651bee9b3dc2ebdc32aefbc6dc73e084c89efbbf8fc6a310a49214ee308f74" exitCode=60 Nov 03 13:45:04.027046 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.026967 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" event={"ID":"091370ad-07c4-41c9-99e5-e990c02eb67a","Type":"ContainerDied","Data":"81651bee9b3dc2ebdc32aefbc6dc73e084c89efbbf8fc6a310a49214ee308f74"} Nov 03 13:45:04.027153 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.027140 4541 scope.go:122] "RemoveContainer" containerID="81651bee9b3dc2ebdc32aefbc6dc73e084c89efbbf8fc6a310a49214ee308f74" Nov 03 13:45:04.027383 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:04.027367 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895225-b9x5x_odh-ai-gateway-infra(091370ad-07c4-41c9-99e5-e990c02eb67a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" Nov 03 13:45:04.036035 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.036013 4541 scope.go:122] "RemoveContainer" containerID="17cc214c805e402c6a5e0ab36fd579df057e436fab61506006e7bd990072249c" Nov 03 13:45:04.645977 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.645921 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:45:04.746579 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.746543 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istiod-ca-cert\") pod \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " Nov 03 13:45:04.746757 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.746600 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-credential-socket\") pod \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " Nov 03 13:45:04.746757 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.746720 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-token\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-token\") pod \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " Nov 03 13:45:04.746906 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.746798 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-certs\") pod \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " Nov 03 13:45:04.746964 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.746914 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-envoy\") pod \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " Nov 03 13:45:04.747021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.746976 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-kube-api-access-fd2zh\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-kube-api-access-fd2zh\") pod \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " Nov 03 13:45:04.747021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747006 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-socket\") pod \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " Nov 03 13:45:04.747120 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747040 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-data\") pod \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " Nov 03 13:45:04.747120 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747072 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-podinfo\") pod \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\" (UID: \"aaccd7db-1711-4a3f-9ef5-cf0624d88b22\") " Nov 03 13:45:04.747120 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.746911 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-credential-socket" pod "aaccd7db-1711-4a3f-9ef5-cf0624d88b22" (UID: "aaccd7db-1711-4a3f-9ef5-cf0624d88b22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:04.747279 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747236 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-data" pod "aaccd7db-1711-4a3f-9ef5-cf0624d88b22" (UID: "aaccd7db-1711-4a3f-9ef5-cf0624d88b22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:04.747279 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.746942 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istiod-ca-cert" pod "aaccd7db-1711-4a3f-9ef5-cf0624d88b22" (UID: "aaccd7db-1711-4a3f-9ef5-cf0624d88b22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:04.747279 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747065 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-certs" pod "aaccd7db-1711-4a3f-9ef5-cf0624d88b22" (UID: "aaccd7db-1711-4a3f-9ef5-cf0624d88b22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:04.747426 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747277 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-socket" pod "aaccd7db-1711-4a3f-9ef5-cf0624d88b22" (UID: "aaccd7db-1711-4a3f-9ef5-cf0624d88b22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:04.747563 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747540 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:04.747563 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747564 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:04.747743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747578 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:04.747743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747592 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:04.747743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.747604 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:04.748999 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.748971 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-envoy" pod "aaccd7db-1711-4a3f-9ef5-cf0624d88b22" (UID: "aaccd7db-1711-4a3f-9ef5-cf0624d88b22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:04.749134 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.749007 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-token" pod "aaccd7db-1711-4a3f-9ef5-cf0624d88b22" (UID: "aaccd7db-1711-4a3f-9ef5-cf0624d88b22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:04.749428 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.749403 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-kube-api-access-fd2zh" pod "aaccd7db-1711-4a3f-9ef5-cf0624d88b22" (UID: "aaccd7db-1711-4a3f-9ef5-cf0624d88b22"). InnerVolumeSpecName "kube-api-access-fd2zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:04.749428 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.749409 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-podinfo" pod "aaccd7db-1711-4a3f-9ef5-cf0624d88b22" (UID: "aaccd7db-1711-4a3f-9ef5-cf0624d88b22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:45:04.848256 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.848221 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:04.848256 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.848252 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:04.848256 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.848262 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd2zh\" (UniqueName: \"kubernetes.io/projected/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-kube-api-access-fd2zh\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:04.848502 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.848281 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaccd7db-1711-4a3f-9ef5-cf0624d88b22-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:04.868067 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:04.868032 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" path="/var/lib/kubelet/pods/cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1/volumes" Nov 03 13:45:05.035846 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.035812 4541 scope.go:122] "RemoveContainer" containerID="5757a09dd5313ddc1507fb1b27fa2113f3eabff5e4457552c40963a7e1da52e8" Nov 03 13:45:05.036289 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:05.036103 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn_odh-ai-gateway-infra(74d9fbf2-bde0-4801-8e6c-3882553894f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" Nov 03 13:45:05.037434 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.037413 4541 scope.go:122] "RemoveContainer" containerID="81651bee9b3dc2ebdc32aefbc6dc73e084c89efbbf8fc6a310a49214ee308f74" Nov 03 13:45:05.037677 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:05.037656 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895225-b9x5x_odh-ai-gateway-infra(091370ad-07c4-41c9-99e5-e990c02eb67a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" Nov 03 13:45:05.038441 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.038418 4541 generic.go:356] "Generic (PLEG): container finished" podID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerID="3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941" exitCode=0 Nov 03 13:45:05.038534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.038451 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" Nov 03 13:45:05.038534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.038457 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" event={"ID":"aaccd7db-1711-4a3f-9ef5-cf0624d88b22","Type":"ContainerDied","Data":"3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941"} Nov 03 13:45:05.038534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.038498 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf" event={"ID":"aaccd7db-1711-4a3f-9ef5-cf0624d88b22","Type":"ContainerDied","Data":"35f74d2ef446cbcc1ecddcb7660b9877e643c4f8774cbdbaecc7c7fc6c238d9a"} Nov 03 13:45:05.038534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.038521 4541 scope.go:122] "RemoveContainer" containerID="3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941" Nov 03 13:45:05.047462 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.047442 4541 scope.go:122] "RemoveContainer" containerID="3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941" Nov 03 13:45:05.047747 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:05.047725 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941\": container with ID starting with 3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941 not found: ID does not exist" containerID="3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941" Nov 03 13:45:05.047842 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.047755 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941"} err="failed to get container status \"3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941\": rpc error: code = NotFound desc = could not find container \"3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941\": container with ID starting with 3aab49c2ff92f120c7f75a1207c749eb27d1cbf28f6495ed06bfb921fbd82941 not found: ID does not exist" Nov 03 13:45:05.270247 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.270214 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf"] Nov 03 13:45:05.275833 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:05.275803 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-z88pf"] Nov 03 13:45:06.082232 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.082194 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8"] Nov 03 13:45:06.082926 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.082905 4541 cpu_manager.go:414] "containerMap: removing container" podUID="cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" containerName="discovery" Nov 03 13:45:06.082926 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.082926 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" containerName="discovery" Nov 03 13:45:06.083100 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.082957 4541 cpu_manager.go:414] "containerMap: removing container" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" Nov 03 13:45:06.083100 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.082965 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" Nov 03 13:45:06.083206 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.083105 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbdff1fd-bfbf-453d-97b2-6ee9158cf0c1" containerName="discovery" Nov 03 13:45:06.083206 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.083123 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" containerName="istio-proxy" Nov 03 13:45:06.087936 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.087917 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.138106 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.138074 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:45:06.151215 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.151190 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8"] Nov 03 13:45:06.162278 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.162252 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98f3f387-ggtt8\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.162429 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.162303 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98f3f387-ggtt8\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.162429 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.162385 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhdlc\" (UniqueName: \"kubernetes.io/projected/34b86734-8ee2-41b5-a17b-6c9924345b89-kube-api-access-xhdlc\") pod \"maas-api-revoke-sub-98f3f387-ggtt8\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.263418 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.263375 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhdlc\" (UniqueName: \"kubernetes.io/projected/34b86734-8ee2-41b5-a17b-6c9924345b89-kube-api-access-xhdlc\") pod \"maas-api-revoke-sub-98f3f387-ggtt8\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.263586 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.263549 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98f3f387-ggtt8\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.263640 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.263614 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98f3f387-ggtt8\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.264272 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.264242 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98f3f387-ggtt8\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.265932 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.265915 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98f3f387-ggtt8\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.319910 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.319879 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhdlc\" (UniqueName: \"kubernetes.io/projected/34b86734-8ee2-41b5-a17b-6c9924345b89-kube-api-access-xhdlc\") pod \"maas-api-revoke-sub-98f3f387-ggtt8\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.400883 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.400853 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:06.794027 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.792744 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8"] Nov 03 13:45:06.797691 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:45:06.797658 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b86734_8ee2_41b5_a17b_6c9924345b89.slice/crio-4012067cdafefb1b9581cba37306749a007b0e7e754c0f2d2baceaef57dadd09 WatchSource:0}: Error finding container 4012067cdafefb1b9581cba37306749a007b0e7e754c0f2d2baceaef57dadd09: Status 404 returned error can't find the container with id 4012067cdafefb1b9581cba37306749a007b0e7e754c0f2d2baceaef57dadd09 Nov 03 13:45:06.868234 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:06.868205 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaccd7db-1711-4a3f-9ef5-cf0624d88b22" path="/var/lib/kubelet/pods/aaccd7db-1711-4a3f-9ef5-cf0624d88b22/volumes" Nov 03 13:45:07.054397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:07.054364 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" event={"ID":"34b86734-8ee2-41b5-a17b-6c9924345b89","Type":"ContainerStarted","Data":"0a365ae9062a7ee4670fc6d9d96c83743a39bdd2d233deec85060cb3db20a8d4"} Nov 03 13:45:07.054493 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:07.054412 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" event={"ID":"34b86734-8ee2-41b5-a17b-6c9924345b89","Type":"ContainerStarted","Data":"4012067cdafefb1b9581cba37306749a007b0e7e754c0f2d2baceaef57dadd09"} Nov 03 13:45:07.170811 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:07.170739 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" podStartSLOduration=2.170725088 podStartE2EDuration="2.170725088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:07.168545739 +0000 UTC m=+2438.482517602" watchObservedRunningTime="2026-11-03 13:45:07.170725088 +0000 UTC m=+2438.484696947" Nov 03 13:45:08.061900 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:08.061869 4541 generic.go:356] "Generic (PLEG): container finished" podID="34b86734-8ee2-41b5-a17b-6c9924345b89" containerID="0a365ae9062a7ee4670fc6d9d96c83743a39bdd2d233deec85060cb3db20a8d4" exitCode=0 Nov 03 13:45:08.062083 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:08.061958 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" event={"ID":"34b86734-8ee2-41b5-a17b-6c9924345b89","Type":"ContainerDied","Data":"0a365ae9062a7ee4670fc6d9d96c83743a39bdd2d233deec85060cb3db20a8d4"} Nov 03 13:45:09.206703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.206677 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:09.297088 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.297051 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b86734-8ee2-41b5-a17b-6c9924345b89-kube-api-access-xhdlc\" (UniqueName: \"kubernetes.io/projected/34b86734-8ee2-41b5-a17b-6c9924345b89-kube-api-access-xhdlc\") pod \"34b86734-8ee2-41b5-a17b-6c9924345b89\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " Nov 03 13:45:09.297282 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.297203 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-xks-ca\") pod \"34b86734-8ee2-41b5-a17b-6c9924345b89\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " Nov 03 13:45:09.297282 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.297252 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-openshift-ca\") pod \"34b86734-8ee2-41b5-a17b-6c9924345b89\" (UID: \"34b86734-8ee2-41b5-a17b-6c9924345b89\") " Nov 03 13:45:09.297614 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.297592 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-openshift-ca" pod "34b86734-8ee2-41b5-a17b-6c9924345b89" (UID: "34b86734-8ee2-41b5-a17b-6c9924345b89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:09.299133 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.299111 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b86734-8ee2-41b5-a17b-6c9924345b89-kube-api-access-xhdlc" pod "34b86734-8ee2-41b5-a17b-6c9924345b89" (UID: "34b86734-8ee2-41b5-a17b-6c9924345b89"). InnerVolumeSpecName "kube-api-access-xhdlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:09.299208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.299130 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-xks-ca" pod "34b86734-8ee2-41b5-a17b-6c9924345b89" (UID: "34b86734-8ee2-41b5-a17b-6c9924345b89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:09.398512 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.398484 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhdlc\" (UniqueName: \"kubernetes.io/projected/34b86734-8ee2-41b5-a17b-6c9924345b89-kube-api-access-xhdlc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:09.398656 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.398515 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:09.398656 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:09.398525 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b86734-8ee2-41b5-a17b-6c9924345b89-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:10.073867 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:10.073828 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" Nov 03 13:45:10.073867 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:10.073842 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8" event={"ID":"34b86734-8ee2-41b5-a17b-6c9924345b89","Type":"ContainerDied","Data":"4012067cdafefb1b9581cba37306749a007b0e7e754c0f2d2baceaef57dadd09"} Nov 03 13:45:10.074092 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:10.073888 4541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4012067cdafefb1b9581cba37306749a007b0e7e754c0f2d2baceaef57dadd09" Nov 03 13:45:13.862103 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:13.862071 4541 scope.go:122] "RemoveContainer" containerID="5757a09dd5313ddc1507fb1b27fa2113f3eabff5e4457552c40963a7e1da52e8" Nov 03 13:45:15.101676 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:15.101642 4541 generic.go:356] "Generic (PLEG): container finished" podID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerID="6aa41636db1afe60b657cc144278e898a68e4a26b8f4633db7b2c1fd77283675" exitCode=60 Nov 03 13:45:15.102100 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:15.101717 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" event={"ID":"74d9fbf2-bde0-4801-8e6c-3882553894f5","Type":"ContainerDied","Data":"6aa41636db1afe60b657cc144278e898a68e4a26b8f4633db7b2c1fd77283675"} Nov 03 13:45:15.102100 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:15.101806 4541 scope.go:122] "RemoveContainer" containerID="5757a09dd5313ddc1507fb1b27fa2113f3eabff5e4457552c40963a7e1da52e8" Nov 03 13:45:15.102215 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:15.102180 4541 scope.go:122] "RemoveContainer" containerID="6aa41636db1afe60b657cc144278e898a68e4a26b8f4633db7b2c1fd77283675" Nov 03 13:45:15.102467 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:15.102448 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn_odh-ai-gateway-infra(74d9fbf2-bde0-4801-8e6c-3882553894f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" Nov 03 13:45:16.236950 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:16.236920 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn"] Nov 03 13:45:16.375109 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:16.375084 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:16.474102 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:16.474067 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d9fbf2-bde0-4801-8e6c-3882553894f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d9fbf2-bde0-4801-8e6c-3882553894f5-maas-api-service-ca\") pod \"74d9fbf2-bde0-4801-8e6c-3882553894f5\" (UID: \"74d9fbf2-bde0-4801-8e6c-3882553894f5\") " Nov 03 13:45:16.474300 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:16.474170 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d9fbf2-bde0-4801-8e6c-3882553894f5-kube-api-access-q2p28\" (UniqueName: \"kubernetes.io/projected/74d9fbf2-bde0-4801-8e6c-3882553894f5-kube-api-access-q2p28\") pod \"74d9fbf2-bde0-4801-8e6c-3882553894f5\" (UID: \"74d9fbf2-bde0-4801-8e6c-3882553894f5\") " Nov 03 13:45:16.474449 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:16.474425 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d9fbf2-bde0-4801-8e6c-3882553894f5-maas-api-service-ca" pod "74d9fbf2-bde0-4801-8e6c-3882553894f5" (UID: "74d9fbf2-bde0-4801-8e6c-3882553894f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:16.474541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:16.474528 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74d9fbf2-bde0-4801-8e6c-3882553894f5-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:16.476164 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:16.476142 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d9fbf2-bde0-4801-8e6c-3882553894f5-kube-api-access-q2p28" pod "74d9fbf2-bde0-4801-8e6c-3882553894f5" (UID: "74d9fbf2-bde0-4801-8e6c-3882553894f5"). InnerVolumeSpecName "kube-api-access-q2p28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:16.575650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:16.575570 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2p28\" (UniqueName: \"kubernetes.io/projected/74d9fbf2-bde0-4801-8e6c-3882553894f5-kube-api-access-q2p28\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:16.890823 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:16.890786 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:45:17.116079 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:17.116051 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" Nov 03 13:45:17.116079 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:17.116061 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn" event={"ID":"74d9fbf2-bde0-4801-8e6c-3882553894f5","Type":"ContainerDied","Data":"94e8f4d91740d099069960a099a6fd2a414aee4a9264c8f7cc075d2895e74f66"} Nov 03 13:45:17.116324 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:17.116115 4541 scope.go:122] "RemoveContainer" containerID="6aa41636db1afe60b657cc144278e898a68e4a26b8f4633db7b2c1fd77283675" Nov 03 13:45:17.300635 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:17.300595 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn"] Nov 03 13:45:17.347061 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:17.347028 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-32bec3-29895225-b2tgn"] Nov 03 13:45:17.861481 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:17.861452 4541 scope.go:122] "RemoveContainer" containerID="81651bee9b3dc2ebdc32aefbc6dc73e084c89efbbf8fc6a310a49214ee308f74" Nov 03 13:45:18.871401 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:18.871364 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" path="/var/lib/kubelet/pods/74d9fbf2-bde0-4801-8e6c-3882553894f5/volumes" Nov 03 13:45:19.136985 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:19.136901 4541 generic.go:356] "Generic (PLEG): container finished" podID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerID="484fe7188587e21e9ec119b3113c41c9bcaebcd7dfebc5948e8b562324f120bd" exitCode=60 Nov 03 13:45:19.136985 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:19.136940 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" event={"ID":"091370ad-07c4-41c9-99e5-e990c02eb67a","Type":"ContainerDied","Data":"484fe7188587e21e9ec119b3113c41c9bcaebcd7dfebc5948e8b562324f120bd"} Nov 03 13:45:19.137202 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:19.136992 4541 scope.go:122] "RemoveContainer" containerID="81651bee9b3dc2ebdc32aefbc6dc73e084c89efbbf8fc6a310a49214ee308f74" Nov 03 13:45:19.137434 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:19.137411 4541 scope.go:122] "RemoveContainer" containerID="484fe7188587e21e9ec119b3113c41c9bcaebcd7dfebc5948e8b562324f120bd" Nov 03 13:45:19.137820 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:19.137757 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895225-b9x5x_odh-ai-gateway-infra(091370ad-07c4-41c9-99e5-e990c02eb67a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" Nov 03 13:45:20.263267 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:20.263232 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x"] Nov 03 13:45:20.415419 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:20.415395 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:20.514873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:20.514767 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091370ad-07c4-41c9-99e5-e990c02eb67a-kube-api-access-59xxc\" (UniqueName: \"kubernetes.io/projected/091370ad-07c4-41c9-99e5-e990c02eb67a-kube-api-access-59xxc\") pod \"091370ad-07c4-41c9-99e5-e990c02eb67a\" (UID: \"091370ad-07c4-41c9-99e5-e990c02eb67a\") " Nov 03 13:45:20.515020 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:20.514893 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091370ad-07c4-41c9-99e5-e990c02eb67a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091370ad-07c4-41c9-99e5-e990c02eb67a-maas-api-service-ca\") pod \"091370ad-07c4-41c9-99e5-e990c02eb67a\" (UID: \"091370ad-07c4-41c9-99e5-e990c02eb67a\") " Nov 03 13:45:20.515328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:20.515297 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091370ad-07c4-41c9-99e5-e990c02eb67a-maas-api-service-ca" pod "091370ad-07c4-41c9-99e5-e990c02eb67a" (UID: "091370ad-07c4-41c9-99e5-e990c02eb67a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:20.516960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:20.516940 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091370ad-07c4-41c9-99e5-e990c02eb67a-kube-api-access-59xxc" pod "091370ad-07c4-41c9-99e5-e990c02eb67a" (UID: "091370ad-07c4-41c9-99e5-e990c02eb67a"). InnerVolumeSpecName "kube-api-access-59xxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:20.616702 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:20.616663 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59xxc\" (UniqueName: \"kubernetes.io/projected/091370ad-07c4-41c9-99e5-e990c02eb67a-kube-api-access-59xxc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:20.616702 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:20.616698 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/091370ad-07c4-41c9-99e5-e990c02eb67a-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:21.151248 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:21.151218 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" event={"ID":"091370ad-07c4-41c9-99e5-e990c02eb67a","Type":"ContainerDied","Data":"8ad34e6789fad2850821c84c7b0d21facbc41f5ac037e1f91cb3d5fe71b5d3b7"} Nov 03 13:45:21.151433 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:21.151249 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x" Nov 03 13:45:21.151433 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:21.151286 4541 scope.go:122] "RemoveContainer" containerID="484fe7188587e21e9ec119b3113c41c9bcaebcd7dfebc5948e8b562324f120bd" Nov 03 13:45:21.315622 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:21.315589 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x"] Nov 03 13:45:21.322595 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:21.322567 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895225-b9x5x"] Nov 03 13:45:22.868308 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:22.868257 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" path="/var/lib/kubelet/pods/091370ad-07c4-41c9-99e5-e990c02eb67a/volumes" Nov 03 13:45:42.957092 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:42.957012 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj"] Nov 03 13:45:43.058226 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:43.058188 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586"] Nov 03 13:45:46.890137 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:46.890101 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:45:47.958379 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:47.958311 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" podUID="55438f58-edb0-41ba-bc20-0abcd94dfc21" containerName="payload-processing" containerID="cri-o://b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45" gracePeriod=25 Nov 03 13:45:48.059384 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.059339 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" podUID="b98100ef-ccee-4d2d-af02-9f2d8cd87289" containerName="payload-pre-processing" containerID="cri-o://e129479b9858b2bcc50b811613444fdd0847feb183099090b96d75e87d86d80e" gracePeriod=25 Nov 03 13:45:48.222688 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.222614 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:45:48.303995 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.303954 4541 generic.go:356] "Generic (PLEG): container finished" podID="55438f58-edb0-41ba-bc20-0abcd94dfc21" containerID="b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45" exitCode=0 Nov 03 13:45:48.304201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.304032 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" Nov 03 13:45:48.304201 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.304028 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" event={"ID":"55438f58-edb0-41ba-bc20-0abcd94dfc21","Type":"ContainerDied","Data":"b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45"} Nov 03 13:45:48.304316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.304217 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj" event={"ID":"55438f58-edb0-41ba-bc20-0abcd94dfc21","Type":"ContainerDied","Data":"483185632a73ecfe5fbfd71392e416317ffc7fa1713bd0ac6565b1e53482666c"} Nov 03 13:45:48.304316 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.304249 4541 scope.go:122] "RemoveContainer" containerID="b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45" Nov 03 13:45:48.307008 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.306981 4541 generic.go:356] "Generic (PLEG): container finished" podID="b98100ef-ccee-4d2d-af02-9f2d8cd87289" containerID="e129479b9858b2bcc50b811613444fdd0847feb183099090b96d75e87d86d80e" exitCode=0 Nov 03 13:45:48.307155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.307133 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" event={"ID":"b98100ef-ccee-4d2d-af02-9f2d8cd87289","Type":"ContainerDied","Data":"e129479b9858b2bcc50b811613444fdd0847feb183099090b96d75e87d86d80e"} Nov 03 13:45:48.316638 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.316618 4541 scope.go:122] "RemoveContainer" containerID="b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45" Nov 03 13:45:48.316980 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:45:48.316958 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45\": container with ID starting with b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45 not found: ID does not exist" containerID="b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45" Nov 03 13:45:48.317086 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.316986 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45"} err="failed to get container status \"b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45\": rpc error: code = NotFound desc = could not find container \"b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45\": container with ID starting with b0b1ca7b0fe8c7673a73ad30422e8f80890133d7c3c52cf0de243dd6143b5e45 not found: ID does not exist" Nov 03 13:45:48.319663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.319641 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55438f58-edb0-41ba-bc20-0abcd94dfc21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55438f58-edb0-41ba-bc20-0abcd94dfc21-plugins-config-volume\") pod \"55438f58-edb0-41ba-bc20-0abcd94dfc21\" (UID: \"55438f58-edb0-41ba-bc20-0abcd94dfc21\") " Nov 03 13:45:48.319758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.319677 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55438f58-edb0-41ba-bc20-0abcd94dfc21-kube-api-access-nh2vv\" (UniqueName: \"kubernetes.io/projected/55438f58-edb0-41ba-bc20-0abcd94dfc21-kube-api-access-nh2vv\") pod \"55438f58-edb0-41ba-bc20-0abcd94dfc21\" (UID: \"55438f58-edb0-41ba-bc20-0abcd94dfc21\") " Nov 03 13:45:48.319976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.319954 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55438f58-edb0-41ba-bc20-0abcd94dfc21-plugins-config-volume" pod "55438f58-edb0-41ba-bc20-0abcd94dfc21" (UID: "55438f58-edb0-41ba-bc20-0abcd94dfc21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:48.321758 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.321737 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55438f58-edb0-41ba-bc20-0abcd94dfc21-kube-api-access-nh2vv" pod "55438f58-edb0-41ba-bc20-0abcd94dfc21" (UID: "55438f58-edb0-41ba-bc20-0abcd94dfc21"). InnerVolumeSpecName "kube-api-access-nh2vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:48.348530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.348501 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:45:48.420645 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.420613 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55438f58-edb0-41ba-bc20-0abcd94dfc21-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:48.420645 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.420640 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh2vv\" (UniqueName: \"kubernetes.io/projected/55438f58-edb0-41ba-bc20-0abcd94dfc21-kube-api-access-nh2vv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:48.521153 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.521057 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98100ef-ccee-4d2d-af02-9f2d8cd87289-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98100ef-ccee-4d2d-af02-9f2d8cd87289-plugins-config-volume\") pod \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\" (UID: \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\") " Nov 03 13:45:48.521382 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.521166 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98100ef-ccee-4d2d-af02-9f2d8cd87289-kube-api-access-kqvz7\" (UniqueName: \"kubernetes.io/projected/b98100ef-ccee-4d2d-af02-9f2d8cd87289-kube-api-access-kqvz7\") pod \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\" (UID: \"b98100ef-ccee-4d2d-af02-9f2d8cd87289\") " Nov 03 13:45:48.521517 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.521491 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98100ef-ccee-4d2d-af02-9f2d8cd87289-plugins-config-volume" pod "b98100ef-ccee-4d2d-af02-9f2d8cd87289" (UID: "b98100ef-ccee-4d2d-af02-9f2d8cd87289"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:48.523367 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.523335 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98100ef-ccee-4d2d-af02-9f2d8cd87289-kube-api-access-kqvz7" pod "b98100ef-ccee-4d2d-af02-9f2d8cd87289" (UID: "b98100ef-ccee-4d2d-af02-9f2d8cd87289"). InnerVolumeSpecName "kube-api-access-kqvz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:48.622853 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.622824 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqvz7\" (UniqueName: \"kubernetes.io/projected/b98100ef-ccee-4d2d-af02-9f2d8cd87289-kube-api-access-kqvz7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:48.622853 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.622851 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98100ef-ccee-4d2d-af02-9f2d8cd87289-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:48.721848 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.721809 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj"] Nov 03 13:45:48.727880 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.727853 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-84e22c-58bf49574f-z67qj"] Nov 03 13:45:48.870249 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:48.870167 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55438f58-edb0-41ba-bc20-0abcd94dfc21" path="/var/lib/kubelet/pods/55438f58-edb0-41ba-bc20-0abcd94dfc21/volumes" Nov 03 13:45:49.314669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:49.314642 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" Nov 03 13:45:49.315191 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:49.314667 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586" event={"ID":"b98100ef-ccee-4d2d-af02-9f2d8cd87289","Type":"ContainerDied","Data":"93c8423a109045d6fb28a259eec205300dc190851ce0d5c6d05518297702f415"} Nov 03 13:45:49.315191 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:49.314729 4541 scope.go:122] "RemoveContainer" containerID="e129479b9858b2bcc50b811613444fdd0847feb183099090b96d75e87d86d80e" Nov 03 13:45:49.471731 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:49.471681 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586"] Nov 03 13:45:49.518397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:49.518361 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-84e22c-66f6958467-p8586"] Nov 03 13:45:50.635075 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635040 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9"] Nov 03 13:45:50.635735 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635713 4541 cpu_manager.go:414] "containerMap: removing container" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerName="cleanup" Nov 03 13:45:50.635887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635737 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerName="cleanup" Nov 03 13:45:50.635887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635755 4541 cpu_manager.go:414] "containerMap: removing container" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerName="cleanup" Nov 03 13:45:50.635887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635765 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerName="cleanup" Nov 03 13:45:50.635887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635812 4541 cpu_manager.go:414] "containerMap: removing container" podUID="34b86734-8ee2-41b5-a17b-6c9924345b89" containerName="revoke-keys" Nov 03 13:45:50.635887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635820 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b86734-8ee2-41b5-a17b-6c9924345b89" containerName="revoke-keys" Nov 03 13:45:50.635887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635831 4541 cpu_manager.go:414] "containerMap: removing container" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerName="cleanup" Nov 03 13:45:50.635887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635838 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerName="cleanup" Nov 03 13:45:50.635887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635877 4541 cpu_manager.go:414] "containerMap: removing container" podUID="b98100ef-ccee-4d2d-af02-9f2d8cd87289" containerName="payload-pre-processing" Nov 03 13:45:50.635887 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635885 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98100ef-ccee-4d2d-af02-9f2d8cd87289" containerName="payload-pre-processing" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635903 4541 cpu_manager.go:414] "containerMap: removing container" podUID="55438f58-edb0-41ba-bc20-0abcd94dfc21" containerName="payload-processing" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635911 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55438f58-edb0-41ba-bc20-0abcd94dfc21" containerName="payload-processing" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635935 4541 cpu_manager.go:414] "containerMap: removing container" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerName="cleanup" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.635943 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerName="cleanup" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.636072 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerName="cleanup" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.636087 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98100ef-ccee-4d2d-af02-9f2d8cd87289" containerName="payload-pre-processing" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.636096 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="55438f58-edb0-41ba-bc20-0abcd94dfc21" containerName="payload-processing" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.636109 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerName="cleanup" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.636118 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerName="cleanup" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.636126 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerName="cleanup" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.636136 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerName="cleanup" Nov 03 13:45:50.636360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.636145 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b86734-8ee2-41b5-a17b-6c9924345b89" containerName="revoke-keys" Nov 03 13:45:50.640876 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.640855 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.747558 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.747508 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e510d28-293c-449b-a23e-9e32fe24895b-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.747763 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.747592 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.747763 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.747676 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.747941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.747794 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.747941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.747853 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e510d28-293c-449b-a23e-9e32fe24895b-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.747941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.747896 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.747941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.747926 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz279\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-kube-api-access-sz279\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.748128 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.748002 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.748128 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.748045 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.750894 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.750861 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9"] Nov 03 13:45:50.849344 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849306 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e510d28-293c-449b-a23e-9e32fe24895b-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849344 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849348 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849597 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849398 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849597 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849433 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849597 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849450 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e510d28-293c-449b-a23e-9e32fe24895b-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849760 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849602 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849760 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849648 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz279\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-kube-api-access-sz279\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849760 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849697 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849964 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849814 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849964 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849869 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.849964 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849939 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.850129 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.849983 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.850189 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.850154 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.850363 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.850345 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e510d28-293c-449b-a23e-9e32fe24895b-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.852337 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.852312 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.852575 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.852555 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e510d28-293c-449b-a23e-9e32fe24895b-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.869123 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.869099 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98100ef-ccee-4d2d-af02-9f2d8cd87289" path="/var/lib/kubelet/pods/b98100ef-ccee-4d2d-af02-9f2d8cd87289/volumes" Nov 03 13:45:50.906440 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.906411 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.906650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.906626 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz279\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-kube-api-access-sz279\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:50.954105 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:50.954075 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:51.378429 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:51.378396 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9"] Nov 03 13:45:51.379886 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:45:51.379855 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e510d28_293c_449b_a23e_9e32fe24895b.slice/crio-1c8dd83d084cfc8b2f4565b3f9b3ac3a78fd50dad33eee7d3b84a427d86a24e7 WatchSource:0}: Error finding container 1c8dd83d084cfc8b2f4565b3f9b3ac3a78fd50dad33eee7d3b84a427d86a24e7: Status 404 returned error can't find the container with id 1c8dd83d084cfc8b2f4565b3f9b3ac3a78fd50dad33eee7d3b84a427d86a24e7 Nov 03 13:45:51.382261 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:51.382229 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:45:51.382340 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:51.382297 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:45:51.382340 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:51.382326 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:45:52.359848 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:52.359765 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" event={"ID":"9e510d28-293c-449b-a23e-9e32fe24895b","Type":"ContainerStarted","Data":"2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95"} Nov 03 13:45:52.360284 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:52.359865 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" event={"ID":"9e510d28-293c-449b-a23e-9e32fe24895b","Type":"ContainerStarted","Data":"1c8dd83d084cfc8b2f4565b3f9b3ac3a78fd50dad33eee7d3b84a427d86a24e7"} Nov 03 13:45:52.569407 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:52.569348 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" podStartSLOduration=2.5693301809999998 podStartE2EDuration="2.569330181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:52.522082728 +0000 UTC m=+2483.836054585" watchObservedRunningTime="2026-11-03 13:45:52.569330181 +0000 UTC m=+2483.883302041" Nov 03 13:45:52.958797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:52.958585 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:52.958797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:52.958719 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:45:52.959081 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:52.958798 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 13:45:53.958797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:53.957872 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:45:53.958797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:53.957934 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 13:45:54.959185 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:54.959154 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:45:54.959570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:54.959220 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 13:45:55.955249 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:55.955205 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:45:55.955579 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:55.955547 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 13:45:56.959249 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:56.959210 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:45:56.959715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:56.959271 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 13:45:57.954682 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:57.954642 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 13:45:57.954874 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:57.954714 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 13:45:58.958578 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:58.958547 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:58.959098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:58.959079 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:45:58.959745 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:45:58.959726 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:46:07.615704 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.615665 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk"] Nov 03 13:46:07.616232 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.616216 4541 cpu_manager.go:414] "containerMap: removing container" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerName="cleanup" Nov 03 13:46:07.616302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.616233 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d9fbf2-bde0-4801-8e6c-3882553894f5" containerName="cleanup" Nov 03 13:46:07.616302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.616259 4541 cpu_manager.go:414] "containerMap: removing container" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerName="cleanup" Nov 03 13:46:07.616302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.616273 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerName="cleanup" Nov 03 13:46:07.616401 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.616366 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="091370ad-07c4-41c9-99e5-e990c02eb67a" containerName="cleanup" Nov 03 13:46:07.619844 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.619825 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.671245 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.671215 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-977f32-serving-cert\"" Nov 03 13:46:07.721062 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.721026 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk"] Nov 03 13:46:07.733063 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.733027 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.733226 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.733079 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.733226 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.733163 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjc5\" (UniqueName: \"kubernetes.io/projected/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-kube-api-access-rmjc5\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.733351 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.733323 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-metrics-certs\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.834884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.834846 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.834884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.834884 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjc5\" (UniqueName: \"kubernetes.io/projected/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-kube-api-access-rmjc5\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.835102 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.834954 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-metrics-certs\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.835102 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:46:07.835016 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-serving-cert: secret "maas-api-e2e-praxis-a-977f32-serving-cert" not found Nov 03 13:46:07.835102 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:46:07.835090 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls podName:fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a nodeName:}" failed. No retries permitted until 2026-11-03 13:46:08.335069128 +0000 UTC m=+2499.649040972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls") pod "maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" (UID: "fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a") : secret "maas-api-e2e-praxis-a-977f32-serving-cert" not found Nov 03 13:46:07.835266 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.835154 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.835923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.835901 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.837472 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.837455 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-metrics-certs\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:07.890409 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:07.890331 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjc5\" (UniqueName: \"kubernetes.io/projected/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-kube-api-access-rmjc5\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:08.340118 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:08.340082 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:08.342493 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:08.342469 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:08.533534 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:08.533504 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:08.901611 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:08.901583 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk"] Nov 03 13:46:08.903406 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:46:08.903377 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1aba03_7251_4bfe_a5d1_e2d1f8ef023a.slice/crio-404e19aa53757fa5aeaf02eb551de64b1cf2712501c29aa13e99270dcdbb8677 WatchSource:0}: Error finding container 404e19aa53757fa5aeaf02eb551de64b1cf2712501c29aa13e99270dcdbb8677: Status 404 returned error can't find the container with id 404e19aa53757fa5aeaf02eb551de64b1cf2712501c29aa13e99270dcdbb8677 Nov 03 13:46:08.905016 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:08.904997 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:46:09.461741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:09.461704 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" event={"ID":"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a","Type":"ContainerStarted","Data":"404e19aa53757fa5aeaf02eb551de64b1cf2712501c29aa13e99270dcdbb8677"} Nov 03 13:46:10.472237 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:10.472199 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" event={"ID":"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a","Type":"ContainerStarted","Data":"3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a"} Nov 03 13:46:10.472630 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:10.472349 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:10.582051 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:10.581988 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" podStartSLOduration=2.460557151 podStartE2EDuration="3.581971391s" totalImagesPullingTime="1.12141424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:10.581139254 +0000 UTC m=+2501.895111118" watchObservedRunningTime="2026-11-03 13:46:10.581971391 +0000 UTC m=+2501.895943261" Nov 03 13:46:16.482990 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:16.482957 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:46:16.889546 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:46:16.889463 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:46:20.776879 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:20.776843 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv"] Nov 03 13:46:20.777307 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:20.777044 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" containerID="cri-o://85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262" gracePeriod=30 Nov 03 13:46:25.037166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:25.037123 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt"] Nov 03 13:46:25.037669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:25.037407 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" containerID="cri-o://797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc" gracePeriod=30 Nov 03 13:46:26.042160 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.042087 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:46:26.135432 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135394 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-data\") pod \"363b2c47-22f2-4305-93ab-83b8680d1e01\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " Nov 03 13:46:26.135432 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135448 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-token\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-token\") pod \"363b2c47-22f2-4305-93ab-83b8680d1e01\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " Nov 03 13:46:26.135715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135512 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-envoy\") pod \"363b2c47-22f2-4305-93ab-83b8680d1e01\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " Nov 03 13:46:26.135715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135564 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-socket\") pod \"363b2c47-22f2-4305-93ab-83b8680d1e01\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " Nov 03 13:46:26.135715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135642 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-credential-socket\") pod \"363b2c47-22f2-4305-93ab-83b8680d1e01\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " Nov 03 13:46:26.135715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135653 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-data" pod "363b2c47-22f2-4305-93ab-83b8680d1e01" (UID: "363b2c47-22f2-4305-93ab-83b8680d1e01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:26.135715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135694 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363b2c47-22f2-4305-93ab-83b8680d1e01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363b2c47-22f2-4305-93ab-83b8680d1e01-istiod-ca-cert\") pod \"363b2c47-22f2-4305-93ab-83b8680d1e01\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " Nov 03 13:46:26.135960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135721 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-certs\") pod \"363b2c47-22f2-4305-93ab-83b8680d1e01\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " Nov 03 13:46:26.135960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135794 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-podinfo\") pod \"363b2c47-22f2-4305-93ab-83b8680d1e01\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " Nov 03 13:46:26.135960 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135834 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-kube-api-access-xngj6\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-kube-api-access-xngj6\") pod \"363b2c47-22f2-4305-93ab-83b8680d1e01\" (UID: \"363b2c47-22f2-4305-93ab-83b8680d1e01\") " Nov 03 13:46:26.136097 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.135964 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-credential-socket" pod "363b2c47-22f2-4305-93ab-83b8680d1e01" (UID: "363b2c47-22f2-4305-93ab-83b8680d1e01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:26.136151 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.136097 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-certs" pod "363b2c47-22f2-4305-93ab-83b8680d1e01" (UID: "363b2c47-22f2-4305-93ab-83b8680d1e01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:26.136306 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.136281 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363b2c47-22f2-4305-93ab-83b8680d1e01-istiod-ca-cert" pod "363b2c47-22f2-4305-93ab-83b8680d1e01" (UID: "363b2c47-22f2-4305-93ab-83b8680d1e01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:26.136567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.136412 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.136567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.136429 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363b2c47-22f2-4305-93ab-83b8680d1e01-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.136567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.136443 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.136567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.136456 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.136791 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.136746 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-socket" pod "363b2c47-22f2-4305-93ab-83b8680d1e01" (UID: "363b2c47-22f2-4305-93ab-83b8680d1e01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:26.138341 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.138312 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-envoy" pod "363b2c47-22f2-4305-93ab-83b8680d1e01" (UID: "363b2c47-22f2-4305-93ab-83b8680d1e01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:26.138448 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.138429 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-token" pod "363b2c47-22f2-4305-93ab-83b8680d1e01" (UID: "363b2c47-22f2-4305-93ab-83b8680d1e01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:26.138665 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.138637 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-podinfo" pod "363b2c47-22f2-4305-93ab-83b8680d1e01" (UID: "363b2c47-22f2-4305-93ab-83b8680d1e01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:46:26.138822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.138681 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-kube-api-access-xngj6" pod "363b2c47-22f2-4305-93ab-83b8680d1e01" (UID: "363b2c47-22f2-4305-93ab-83b8680d1e01"). InnerVolumeSpecName "kube-api-access-xngj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:26.200141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.200099 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:46:26.200323 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.200172 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:46:26.237725 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.237684 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.237725 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.237726 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.237952 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.237743 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363b2c47-22f2-4305-93ab-83b8680d1e01-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.237952 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.237757 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363b2c47-22f2-4305-93ab-83b8680d1e01-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.237952 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.237788 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xngj6\" (UniqueName: \"kubernetes.io/projected/363b2c47-22f2-4305-93ab-83b8680d1e01-kube-api-access-xngj6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:26.568860 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.568815 4541 generic.go:356] "Generic (PLEG): container finished" podID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerID="85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262" exitCode=0 Nov 03 13:46:26.569036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.568869 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" Nov 03 13:46:26.569036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.568909 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" event={"ID":"363b2c47-22f2-4305-93ab-83b8680d1e01","Type":"ContainerDied","Data":"85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262"} Nov 03 13:46:26.569036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.568962 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv" event={"ID":"363b2c47-22f2-4305-93ab-83b8680d1e01","Type":"ContainerDied","Data":"0ab98afbef1f577f9ebad46dff1b2679c4270401bced1177a4413b50b7d69f94"} Nov 03 13:46:26.569036 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.568985 4541 scope.go:122] "RemoveContainer" containerID="85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262" Nov 03 13:46:26.579505 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.579486 4541 scope.go:122] "RemoveContainer" containerID="85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262" Nov 03 13:46:26.579829 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:46:26.579808 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262\": container with ID starting with 85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262 not found: ID does not exist" containerID="85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262" Nov 03 13:46:26.579904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.579839 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262"} err="failed to get container status \"85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262\": rpc error: code = NotFound desc = could not find container \"85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262\": container with ID starting with 85a4f411c178d2b50f5000fab99ab9fa504904bb9a0845196669d0a1f2ee7262 not found: ID does not exist" Nov 03 13:46:26.725897 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.725858 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv"] Nov 03 13:46:26.772801 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.772750 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-69xzv"] Nov 03 13:46:26.868259 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:26.868176 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" path="/var/lib/kubelet/pods/363b2c47-22f2-4305-93ab-83b8680d1e01/volumes" Nov 03 13:46:30.304194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.304129 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:46:30.376451 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376417 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c7096b-66b8-4e64-aec0-fa83816adbab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7096b-66b8-4e64-aec0-fa83816adbab-istiod-ca-cert\") pod \"a5c7096b-66b8-4e64-aec0-fa83816adbab\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " Nov 03 13:46:30.376644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376464 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-kube-api-access-ds9pr\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-kube-api-access-ds9pr\") pod \"a5c7096b-66b8-4e64-aec0-fa83816adbab\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " Nov 03 13:46:30.376644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376499 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-credential-socket\") pod \"a5c7096b-66b8-4e64-aec0-fa83816adbab\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " Nov 03 13:46:30.376644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376519 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-token\") pod \"a5c7096b-66b8-4e64-aec0-fa83816adbab\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " Nov 03 13:46:30.376644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376539 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-podinfo\") pod \"a5c7096b-66b8-4e64-aec0-fa83816adbab\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " Nov 03 13:46:30.376644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376619 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-socket\") pod \"a5c7096b-66b8-4e64-aec0-fa83816adbab\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " Nov 03 13:46:30.376644 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376644 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-envoy\") pod \"a5c7096b-66b8-4e64-aec0-fa83816adbab\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " Nov 03 13:46:30.376993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376698 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-data\") pod \"a5c7096b-66b8-4e64-aec0-fa83816adbab\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " Nov 03 13:46:30.376993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376755 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-certs\") pod \"a5c7096b-66b8-4e64-aec0-fa83816adbab\" (UID: \"a5c7096b-66b8-4e64-aec0-fa83816adbab\") " Nov 03 13:46:30.376993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376837 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-credential-socket" pod "a5c7096b-66b8-4e64-aec0-fa83816adbab" (UID: "a5c7096b-66b8-4e64-aec0-fa83816adbab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:30.376993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376951 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-socket" pod "a5c7096b-66b8-4e64-aec0-fa83816adbab" (UID: "a5c7096b-66b8-4e64-aec0-fa83816adbab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:30.376993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.376979 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c7096b-66b8-4e64-aec0-fa83816adbab-istiod-ca-cert" pod "a5c7096b-66b8-4e64-aec0-fa83816adbab" (UID: "a5c7096b-66b8-4e64-aec0-fa83816adbab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:30.377243 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.377061 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-data" pod "a5c7096b-66b8-4e64-aec0-fa83816adbab" (UID: "a5c7096b-66b8-4e64-aec0-fa83816adbab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:30.377243 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.377160 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-certs" pod "a5c7096b-66b8-4e64-aec0-fa83816adbab" (UID: "a5c7096b-66b8-4e64-aec0-fa83816adbab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:30.377243 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.377172 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5c7096b-66b8-4e64-aec0-fa83816adbab-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:30.377243 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.377190 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:30.377243 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.377205 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:30.377243 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.377216 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:30.378910 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.378883 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-token" pod "a5c7096b-66b8-4e64-aec0-fa83816adbab" (UID: "a5c7096b-66b8-4e64-aec0-fa83816adbab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:30.378996 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.378910 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-podinfo" pod "a5c7096b-66b8-4e64-aec0-fa83816adbab" (UID: "a5c7096b-66b8-4e64-aec0-fa83816adbab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:46:30.378996 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.378983 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-kube-api-access-ds9pr" pod "a5c7096b-66b8-4e64-aec0-fa83816adbab" (UID: "a5c7096b-66b8-4e64-aec0-fa83816adbab"). InnerVolumeSpecName "kube-api-access-ds9pr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:30.379078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.379066 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-envoy" pod "a5c7096b-66b8-4e64-aec0-fa83816adbab" (UID: "a5c7096b-66b8-4e64-aec0-fa83816adbab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:30.478192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.478155 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:30.478192 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.478194 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5c7096b-66b8-4e64-aec0-fa83816adbab-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:30.478379 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.478212 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9pr\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-kube-api-access-ds9pr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:30.478379 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.478226 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:30.478379 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.478240 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5c7096b-66b8-4e64-aec0-fa83816adbab-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:30.590814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.590721 4541 generic.go:356] "Generic (PLEG): container finished" podID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerID="797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc" exitCode=0 Nov 03 13:46:30.590814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.590781 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" Nov 03 13:46:30.590814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.590804 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" event={"ID":"a5c7096b-66b8-4e64-aec0-fa83816adbab","Type":"ContainerDied","Data":"797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc"} Nov 03 13:46:30.591038 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.590857 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt" event={"ID":"a5c7096b-66b8-4e64-aec0-fa83816adbab","Type":"ContainerDied","Data":"271b83388283f7d5b3c9cdf38351900ce50da5e35984fc0c5c78995f8c14e3e4"} Nov 03 13:46:30.591038 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.590879 4541 scope.go:122] "RemoveContainer" containerID="797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc" Nov 03 13:46:30.607283 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.607260 4541 scope.go:122] "RemoveContainer" containerID="797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc" Nov 03 13:46:30.607608 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:46:30.607580 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc\": container with ID starting with 797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc not found: ID does not exist" containerID="797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc" Nov 03 13:46:30.607662 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.607621 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc"} err="failed to get container status \"797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc\": rpc error: code = NotFound desc = could not find container \"797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc\": container with ID starting with 797a27913ed3789f20fc2e11f85c28958b695797a73823890b6d635ec17292cc not found: ID does not exist" Nov 03 13:46:30.749661 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.749624 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt"] Nov 03 13:46:30.753715 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.753688 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-bcrrt"] Nov 03 13:46:30.868395 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:46:30.868313 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" path="/var/lib/kubelet/pods/a5c7096b-66b8-4e64-aec0-fa83816adbab/volumes" Nov 03 13:46:46.889452 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:46:46.889419 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:47:16.890322 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:47:16.890288 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:47:21.008291 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.008241 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9"] Nov 03 13:47:21.010570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.008730 4541 cpu_manager.go:414] "containerMap: removing container" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" Nov 03 13:47:21.010570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.008745 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" Nov 03 13:47:21.010570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.008762 4541 cpu_manager.go:414] "containerMap: removing container" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" Nov 03 13:47:21.010570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.008767 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" Nov 03 13:47:21.010570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.008892 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="363b2c47-22f2-4305-93ab-83b8680d1e01" containerName="istio-proxy" Nov 03 13:47:21.010570 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.008908 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c7096b-66b8-4e64-aec0-fa83816adbab" containerName="istio-proxy" Nov 03 13:47:21.011996 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.011976 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:21.065708 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.065678 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9"] Nov 03 13:47:21.094600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.094565 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab28994-6da6-4c95-9a25-de2cc069aa70-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9\" (UID: \"dab28994-6da6-4c95-9a25-de2cc069aa70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:21.094782 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.094701 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5k96\" (UniqueName: \"kubernetes.io/projected/dab28994-6da6-4c95-9a25-de2cc069aa70-kube-api-access-m5k96\") pod \"payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9\" (UID: \"dab28994-6da6-4c95-9a25-de2cc069aa70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:21.095993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.095964 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-10c688\"" Nov 03 13:47:21.109998 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.109966 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-10c688-dockercfg-hg92w\"" Nov 03 13:47:21.195344 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.195307 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5k96\" (UniqueName: \"kubernetes.io/projected/dab28994-6da6-4c95-9a25-de2cc069aa70-kube-api-access-m5k96\") pod \"payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9\" (UID: \"dab28994-6da6-4c95-9a25-de2cc069aa70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:21.195504 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.195424 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab28994-6da6-4c95-9a25-de2cc069aa70-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9\" (UID: \"dab28994-6da6-4c95-9a25-de2cc069aa70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:21.196204 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.196185 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab28994-6da6-4c95-9a25-de2cc069aa70-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9\" (UID: \"dab28994-6da6-4c95-9a25-de2cc069aa70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:21.249331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.249301 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5k96\" (UniqueName: \"kubernetes.io/projected/dab28994-6da6-4c95-9a25-de2cc069aa70-kube-api-access-m5k96\") pod \"payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9\" (UID: \"dab28994-6da6-4c95-9a25-de2cc069aa70\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:21.325438 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.325365 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:21.689680 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.689651 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9"] Nov 03 13:47:21.691851 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:47:21.691821 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab28994_6da6_4c95_9a25_de2cc069aa70.slice/crio-4d497aa9f3400baa0e6d5dc3d0cc5c2b7aacc17f803a5b12649046d96008230a WatchSource:0}: Error finding container 4d497aa9f3400baa0e6d5dc3d0cc5c2b7aacc17f803a5b12649046d96008230a: Status 404 returned error can't find the container with id 4d497aa9f3400baa0e6d5dc3d0cc5c2b7aacc17f803a5b12649046d96008230a Nov 03 13:47:21.871430 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.871395 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" event={"ID":"dab28994-6da6-4c95-9a25-de2cc069aa70","Type":"ContainerStarted","Data":"a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66"} Nov 03 13:47:21.871601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.871448 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" event={"ID":"dab28994-6da6-4c95-9a25-de2cc069aa70","Type":"ContainerStarted","Data":"4d497aa9f3400baa0e6d5dc3d0cc5c2b7aacc17f803a5b12649046d96008230a"} Nov 03 13:47:21.871601 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.871471 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:21.982342 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:21.982232 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" podStartSLOduration=1.982218187 podStartE2EDuration="1.982218187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:21.979381937 +0000 UTC m=+2573.293353796" watchObservedRunningTime="2026-11-03 13:47:21.982218187 +0000 UTC m=+2573.296190046" Nov 03 13:47:32.880037 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:32.880009 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:47:37.209537 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.209502 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm"] Nov 03 13:47:37.214595 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.214572 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.265441 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.265411 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d83c93-serving-cert\"" Nov 03 13:47:37.315622 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.315594 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm"] Nov 03 13:47:37.357551 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.357525 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj8rp\" (UniqueName: \"kubernetes.io/projected/276a435a-e9f8-4867-884a-e2c45184e26e-kube-api-access-xj8rp\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.357695 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.357577 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276a435a-e9f8-4867-884a-e2c45184e26e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.357695 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.357646 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-metrics-certs\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.357785 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.357703 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.458275 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.458244 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.458431 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.458339 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj8rp\" (UniqueName: \"kubernetes.io/projected/276a435a-e9f8-4867-884a-e2c45184e26e-kube-api-access-xj8rp\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.458431 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.458377 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276a435a-e9f8-4867-884a-e2c45184e26e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.458431 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:47:37.458406 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-serving-cert: secret "maas-api-e2e-praxis-a-d83c93-serving-cert" not found Nov 03 13:47:37.458431 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.458406 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-metrics-certs\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.458644 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:47:37.458471 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls podName:276a435a-e9f8-4867-884a-e2c45184e26e nodeName:}" failed. No retries permitted until 2026-11-03 13:47:37.958452869 +0000 UTC m=+2589.272424726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls") pod "maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" (UID: "276a435a-e9f8-4867-884a-e2c45184e26e") : secret "maas-api-e2e-praxis-a-d83c93-serving-cert" not found Nov 03 13:47:37.459076 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.459052 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276a435a-e9f8-4867-884a-e2c45184e26e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.460877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.460834 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-metrics-certs\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.515941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.515909 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj8rp\" (UniqueName: \"kubernetes.io/projected/276a435a-e9f8-4867-884a-e2c45184e26e-kube-api-access-xj8rp\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.962898 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.962866 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:37.965256 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:37.965226 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:38.128028 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:38.127997 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:38.495508 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:38.495475 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm"] Nov 03 13:47:38.496786 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:47:38.496746 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276a435a_e9f8_4867_884a_e2c45184e26e.slice/crio-7267f8e7191877184f3d097708b70065c63814d18c5377791bb243324cf4f481 WatchSource:0}: Error finding container 7267f8e7191877184f3d097708b70065c63814d18c5377791bb243324cf4f481: Status 404 returned error can't find the container with id 7267f8e7191877184f3d097708b70065c63814d18c5377791bb243324cf4f481 Nov 03 13:47:38.960313 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:38.960277 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" event={"ID":"276a435a-e9f8-4867-884a-e2c45184e26e","Type":"ContainerStarted","Data":"7267f8e7191877184f3d097708b70065c63814d18c5377791bb243324cf4f481"} Nov 03 13:47:39.967928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:39.967884 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" event={"ID":"276a435a-e9f8-4867-884a-e2c45184e26e","Type":"ContainerStarted","Data":"c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4"} Nov 03 13:47:39.968320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:39.967954 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:40.078208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:40.078150 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" podStartSLOduration=2.202535436 podStartE2EDuration="3.078131666s" totalImagesPullingTime="875.59623ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:40.077516494 +0000 UTC m=+2591.391488354" watchObservedRunningTime="2026-11-03 13:47:40.078131666 +0000 UTC m=+2591.392103526" Nov 03 13:47:45.980577 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:45.980543 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:47:46.904925 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:47:46.904887 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:47:51.474686 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.474644 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m"] Nov 03 13:47:51.479045 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.479022 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.506890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.506856 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.507038 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.506986 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.507081 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.507051 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.507116 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.507079 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.507169 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.507145 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.507269 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.507256 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.507303 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.507291 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.507343 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.507311 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftvk\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-kube-api-access-xftvk\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.507388 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.507377 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.559054 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.558998 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m"] Nov 03 13:47:51.607942 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.607900 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j"] Nov 03 13:47:51.608435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608409 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.608483 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608469 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.608527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608493 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.608639 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608616 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.608754 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608740 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.608853 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608830 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.608914 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608863 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftvk\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-kube-api-access-xftvk\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.608914 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608889 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.608914 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608901 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.609071 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608954 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.609071 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.608974 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.609071 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.609013 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.609237 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.609110 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.609523 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.609501 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.611321 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.611296 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.611641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.611622 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.612636 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.612617 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.612982 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.612959 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42c707-openshift-default-dockercfg-7t7mv\"" Nov 03 13:47:51.663322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.663286 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.663503 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.663482 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftvk\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-kube-api-access-xftvk\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.673845 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.673723 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j"] Nov 03 13:47:51.710239 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.710208 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.710448 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.710244 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.710448 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.710294 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.710574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.710440 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.710574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.710477 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.710574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.710507 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.710574 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.710566 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99spt\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-kube-api-access-99spt\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.710814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.710672 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.710814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.710719 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.792426 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.792332 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:51.811328 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811300 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.811467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811334 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.811467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811378 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.811467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811411 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.811467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811426 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.811467 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811441 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.811718 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811630 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99spt\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-kube-api-access-99spt\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.811828 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811799 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.811922 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811852 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.811922 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811802 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.812138 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.811956 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.812138 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.812038 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.812510 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.812460 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.812799 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.812740 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.814685 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.814661 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.814941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.814921 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.866515 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.866480 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.867532 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.867505 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99spt\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-kube-api-access-99spt\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:51.946237 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:51.946187 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:52.209927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:52.209888 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m"] Nov 03 13:47:52.212157 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:47:52.212125 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6fcb72e_f1e8_4e88_ac56_fde05b65847c.slice/crio-47fc4b51baf16d791a961198fcf92c48e5c0028813734dde8691d3b8aa953cd0 WatchSource:0}: Error finding container 47fc4b51baf16d791a961198fcf92c48e5c0028813734dde8691d3b8aa953cd0: Status 404 returned error can't find the container with id 47fc4b51baf16d791a961198fcf92c48e5c0028813734dde8691d3b8aa953cd0 Nov 03 13:47:52.217997 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:52.217954 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.218173 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:52.218152 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.218229 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:52.218195 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.319141 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:47:52.319097 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2fafe5_bc7b_4291_943c_c501430b7cf4.slice/crio-7f5388b570d9d2fdfc0e708c2f1ba554d7bcc6fbe1ccbb58537d9319af3359e4 WatchSource:0}: Error finding container 7f5388b570d9d2fdfc0e708c2f1ba554d7bcc6fbe1ccbb58537d9319af3359e4: Status 404 returned error can't find the container with id 7f5388b570d9d2fdfc0e708c2f1ba554d7bcc6fbe1ccbb58537d9319af3359e4 Nov 03 13:47:52.319537 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:52.319509 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j"] Nov 03 13:47:52.327798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:52.322960 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.327798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:52.323113 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:52.327798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:52.323216 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:47:53.078367 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.078316 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" event={"ID":"b6fcb72e-f1e8-4e88-ac56-fde05b65847c","Type":"ContainerStarted","Data":"6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7"} Nov 03 13:47:53.079004 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.078976 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" event={"ID":"b6fcb72e-f1e8-4e88-ac56-fde05b65847c","Type":"ContainerStarted","Data":"47fc4b51baf16d791a961198fcf92c48e5c0028813734dde8691d3b8aa953cd0"} Nov 03 13:47:53.089144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.089105 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" event={"ID":"ef2fafe5-bc7b-4291-943c-c501430b7cf4","Type":"ContainerStarted","Data":"19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89"} Nov 03 13:47:53.089321 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.089165 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" event={"ID":"ef2fafe5-bc7b-4291-943c-c501430b7cf4","Type":"ContainerStarted","Data":"7f5388b570d9d2fdfc0e708c2f1ba554d7bcc6fbe1ccbb58537d9319af3359e4"} Nov 03 13:47:53.208136 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.208079 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" podStartSLOduration=2.208064471 podStartE2EDuration="2.208064471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:53.204902936 +0000 UTC m=+2604.518874795" watchObservedRunningTime="2026-11-03 13:47:53.208064471 +0000 UTC m=+2604.522036331" Nov 03 13:47:53.322129 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.322077 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" podStartSLOduration=2.322063189 podStartE2EDuration="2.322063189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:53.320260955 +0000 UTC m=+2604.634232815" watchObservedRunningTime="2026-11-03 13:47:53.322063189 +0000 UTC m=+2604.636035048" Nov 03 13:47:53.793054 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.793021 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:53.794375 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.794341 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:47:53.794526 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.794403 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 13:47:53.966796 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.966574 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:53.979842 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.974196 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:47:53.979842 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:53.974265 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 13:47:54.793395 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:54.793355 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:47:54.793842 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:54.793413 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 13:47:54.949797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:54.947305 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:47:54.949797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:54.947369 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 13:47:55.792785 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:55.792741 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:47:55.792982 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:55.792818 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 13:47:55.950234 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:55.950192 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:47:55.950881 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:55.950848 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 13:47:56.793446 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:56.793406 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:47:56.793642 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:56.793476 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 13:47:56.953795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:56.950218 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:47:56.953795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:56.950286 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 13:47:57.793119 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:57.793069 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:47:57.793294 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:57.793138 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 13:47:57.947233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:57.947190 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:47:57.947512 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:57.947482 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 13:47:58.792843 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:58.792799 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:47:58.793239 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:58.792890 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Nov 03 13:47:58.947242 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:58.947205 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 03 13:47:58.947435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:58.947266 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 03 13:47:59.797155 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:59.797126 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:59.797619 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:59.797556 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:59.798331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:59.798311 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:47:59.956987 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:59.956950 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:59.957624 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:59.957522 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:47:59.958575 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:47:59.958551 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:48:08.962262 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:08.962228 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv"] Nov 03 13:48:08.966317 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:08.966298 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.015475 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.015446 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-10c688-serving-cert\"" Nov 03 13:48:09.063000 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.062967 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.063177 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.063157 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-metrics-certs\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.063338 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.063314 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsqh\" (UniqueName: \"kubernetes.io/projected/0b44cba8-006c-4ba8-959e-ff72144a7dec-kube-api-access-mqsqh\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.063406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.063361 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b44cba8-006c-4ba8-959e-ff72144a7dec-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.065607 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.065585 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv"] Nov 03 13:48:09.164302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.164258 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsqh\" (UniqueName: \"kubernetes.io/projected/0b44cba8-006c-4ba8-959e-ff72144a7dec-kube-api-access-mqsqh\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.164498 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.164311 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b44cba8-006c-4ba8-959e-ff72144a7dec-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.164498 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.164340 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.164498 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.164474 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-metrics-certs\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.164645 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:48:09.164537 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-serving-cert: secret "maas-api-e2e-praxis-b-10c688-serving-cert" not found Nov 03 13:48:09.164645 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:48:09.164609 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls podName:0b44cba8-006c-4ba8-959e-ff72144a7dec nodeName:}" failed. No retries permitted until 2026-11-03 13:48:09.664585333 +0000 UTC m=+2620.978557450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls") pod "maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" (UID: "0b44cba8-006c-4ba8-959e-ff72144a7dec") : secret "maas-api-e2e-praxis-b-10c688-serving-cert" not found Nov 03 13:48:09.165031 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.165013 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b44cba8-006c-4ba8-959e-ff72144a7dec-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.167080 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.167055 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-metrics-certs\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.222264 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.222185 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsqh\" (UniqueName: \"kubernetes.io/projected/0b44cba8-006c-4ba8-959e-ff72144a7dec-kube-api-access-mqsqh\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.671284 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.671249 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.673634 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.673612 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls\") pod \"maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:09.879614 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:09.879581 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:10.249152 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:10.249120 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv"] Nov 03 13:48:10.250504 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:48:10.250477 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b44cba8_006c_4ba8_959e_ff72144a7dec.slice/crio-59e7d43b8503eccd99342e3214e1217806b97a4829751b8122c47ceb50ca7ccb WatchSource:0}: Error finding container 59e7d43b8503eccd99342e3214e1217806b97a4829751b8122c47ceb50ca7ccb: Status 404 returned error can't find the container with id 59e7d43b8503eccd99342e3214e1217806b97a4829751b8122c47ceb50ca7ccb Nov 03 13:48:11.198282 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:11.198247 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" event={"ID":"0b44cba8-006c-4ba8-959e-ff72144a7dec","Type":"ContainerStarted","Data":"59e7d43b8503eccd99342e3214e1217806b97a4829751b8122c47ceb50ca7ccb"} Nov 03 13:48:12.205378 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:12.205344 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" event={"ID":"0b44cba8-006c-4ba8-959e-ff72144a7dec","Type":"ContainerStarted","Data":"29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074"} Nov 03 13:48:12.205861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:12.205481 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:12.316765 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:12.316706 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" podStartSLOduration=3.274207298 podStartE2EDuration="4.316692671s" totalImagesPullingTime="1.042485373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:12.315313213 +0000 UTC m=+2623.629285072" watchObservedRunningTime="2026-11-03 13:48:12.316692671 +0000 UTC m=+2623.630664533" Nov 03 13:48:16.890451 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:48:16.890408 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:48:18.217597 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:18.217565 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:48:33.377553 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.377514 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg"] Nov 03 13:48:33.382144 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.382118 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:33.432171 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.432131 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2bceb1\"" Nov 03 13:48:33.436708 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.436664 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg"] Nov 03 13:48:33.479351 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.479316 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2bceb1-dockercfg-8g5j4\"" Nov 03 13:48:33.539190 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.539152 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdbxj\" (UniqueName: \"kubernetes.io/projected/af19ec54-5e4a-402c-ad37-82f779d0cfef-kube-api-access-wdbxj\") pod \"payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg\" (UID: \"af19ec54-5e4a-402c-ad37-82f779d0cfef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:33.539396 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.539293 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af19ec54-5e4a-402c-ad37-82f779d0cfef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg\" (UID: \"af19ec54-5e4a-402c-ad37-82f779d0cfef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:33.640862 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.640729 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af19ec54-5e4a-402c-ad37-82f779d0cfef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg\" (UID: \"af19ec54-5e4a-402c-ad37-82f779d0cfef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:33.641046 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.640927 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdbxj\" (UniqueName: \"kubernetes.io/projected/af19ec54-5e4a-402c-ad37-82f779d0cfef-kube-api-access-wdbxj\") pod \"payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg\" (UID: \"af19ec54-5e4a-402c-ad37-82f779d0cfef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:33.641527 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.641500 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af19ec54-5e4a-402c-ad37-82f779d0cfef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg\" (UID: \"af19ec54-5e4a-402c-ad37-82f779d0cfef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:33.697457 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.697422 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdbxj\" (UniqueName: \"kubernetes.io/projected/af19ec54-5e4a-402c-ad37-82f779d0cfef-kube-api-access-wdbxj\") pod \"payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg\" (UID: \"af19ec54-5e4a-402c-ad37-82f779d0cfef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:33.994957 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:33.994923 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:34.363915 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:34.363884 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg"] Nov 03 13:48:34.364828 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:48:34.364764 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf19ec54_5e4a_402c_ad37_82f779d0cfef.slice/crio-8f2d46800f5e58c3319f02b281d862890a6aa6cfdc9d604cc7d9700c786e7cf9 WatchSource:0}: Error finding container 8f2d46800f5e58c3319f02b281d862890a6aa6cfdc9d604cc7d9700c786e7cf9: Status 404 returned error can't find the container with id 8f2d46800f5e58c3319f02b281d862890a6aa6cfdc9d604cc7d9700c786e7cf9 Nov 03 13:48:35.336332 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:35.336291 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" event={"ID":"af19ec54-5e4a-402c-ad37-82f779d0cfef","Type":"ContainerStarted","Data":"9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7"} Nov 03 13:48:35.336797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:35.336351 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" event={"ID":"af19ec54-5e4a-402c-ad37-82f779d0cfef","Type":"ContainerStarted","Data":"8f2d46800f5e58c3319f02b281d862890a6aa6cfdc9d604cc7d9700c786e7cf9"} Nov 03 13:48:35.336797 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:35.336464 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:35.445165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:35.445102 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" podStartSLOduration=2.445088545 podStartE2EDuration="2.445088545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:35.443351212 +0000 UTC m=+2646.757323081" watchObservedRunningTime="2026-11-03 13:48:35.445088545 +0000 UTC m=+2646.759060404" Nov 03 13:48:46.345172 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:46.345141 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 13:48:46.897741 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:48:46.897699 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:48:49.610054 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.610015 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc"] Nov 03 13:48:49.615651 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.615624 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:48:49.668055 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.668018 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc"] Nov 03 13:48:49.711114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.711079 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5faf5d\"" Nov 03 13:48:49.711114 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.711096 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5faf5d-dockercfg-z6jbs\"" Nov 03 13:48:49.725523 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.725483 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7kr7\" (UniqueName: \"kubernetes.io/projected/dcea7270-5708-4b70-b0dd-15304574e276-kube-api-access-l7kr7\") pod \"payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc\" (UID: \"dcea7270-5708-4b70-b0dd-15304574e276\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:48:49.725678 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.725660 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea7270-5708-4b70-b0dd-15304574e276-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc\" (UID: \"dcea7270-5708-4b70-b0dd-15304574e276\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:48:49.826449 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.826409 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7kr7\" (UniqueName: \"kubernetes.io/projected/dcea7270-5708-4b70-b0dd-15304574e276-kube-api-access-l7kr7\") pod \"payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc\" (UID: \"dcea7270-5708-4b70-b0dd-15304574e276\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:48:49.826627 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.826505 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea7270-5708-4b70-b0dd-15304574e276-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc\" (UID: \"dcea7270-5708-4b70-b0dd-15304574e276\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:48:49.827228 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.827206 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea7270-5708-4b70-b0dd-15304574e276-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc\" (UID: \"dcea7270-5708-4b70-b0dd-15304574e276\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:48:49.880638 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.880572 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7kr7\" (UniqueName: \"kubernetes.io/projected/dcea7270-5708-4b70-b0dd-15304574e276-kube-api-access-l7kr7\") pod \"payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc\" (UID: \"dcea7270-5708-4b70-b0dd-15304574e276\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:48:49.932419 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:49.932376 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:48:50.298879 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:50.298849 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc"] Nov 03 13:48:50.300159 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:48:50.300130 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcea7270_5708_4b70_b0dd_15304574e276.slice/crio-c5ff9a7121d7718f4eb1ad6616c1345df78fb91acd9eb6ce4dbcf89049be436c WatchSource:0}: Error finding container c5ff9a7121d7718f4eb1ad6616c1345df78fb91acd9eb6ce4dbcf89049be436c: Status 404 returned error can't find the container with id c5ff9a7121d7718f4eb1ad6616c1345df78fb91acd9eb6ce4dbcf89049be436c Nov 03 13:48:50.420208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:50.420170 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" event={"ID":"dcea7270-5708-4b70-b0dd-15304574e276","Type":"ContainerStarted","Data":"83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121"} Nov 03 13:48:50.420428 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:50.420371 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:48:50.420428 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:50.420400 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" event={"ID":"dcea7270-5708-4b70-b0dd-15304574e276","Type":"ContainerStarted","Data":"c5ff9a7121d7718f4eb1ad6616c1345df78fb91acd9eb6ce4dbcf89049be436c"} Nov 03 13:48:50.529206 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:50.529151 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" podStartSLOduration=1.529136539 podStartE2EDuration="1.529136539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:50.527581428 +0000 UTC m=+2661.841553290" watchObservedRunningTime="2026-11-03 13:48:50.529136539 +0000 UTC m=+2661.843108398" Nov 03 13:48:51.945405 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:51.945373 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2"] Nov 03 13:48:51.949396 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:51.949377 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.059001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.058962 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.059206 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.059082 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.059206 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.059126 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.059332 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.059210 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.059332 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.059261 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.059435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.059353 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.059435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.059392 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-kube-api-access-6nkjq\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.059435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.059423 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.059582 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.059491 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.148416 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.148332 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2"] Nov 03 13:48:52.161562 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.161516 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.161759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.161589 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.161759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.161622 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.161759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.161662 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.161759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.161709 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.162044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.161814 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.162044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.161851 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-kube-api-access-6nkjq\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.162044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.161883 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.162044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.161955 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.162474 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.162450 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.162967 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.162944 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.163183 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.163162 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.163232 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.163213 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.163434 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.163416 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.165663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.165640 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.166322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.166299 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.217343 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.217273 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.217476 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.217455 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-kube-api-access-6nkjq\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.262663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.262630 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:52.685465 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.685427 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2"] Nov 03 13:48:52.686198 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:48:52.686166 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d505a0_416c_4607_a9f4_58c4a3916f1b.slice/crio-df71904e1644984fb64b7a9714cedcbaa49bf60df39b8f9aef832199af1a9a98 WatchSource:0}: Error finding container df71904e1644984fb64b7a9714cedcbaa49bf60df39b8f9aef832199af1a9a98: Status 404 returned error can't find the container with id df71904e1644984fb64b7a9714cedcbaa49bf60df39b8f9aef832199af1a9a98 Nov 03 13:48:52.688853 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.688822 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:52.688941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.688891 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:52.688941 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:52.688920 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:48:53.465489 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:53.465443 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" event={"ID":"a8d505a0-416c-4607-a9f4-58c4a3916f1b","Type":"ContainerStarted","Data":"3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408"} Nov 03 13:48:53.466029 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:53.465602 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" event={"ID":"a8d505a0-416c-4607-a9f4-58c4a3916f1b","Type":"ContainerStarted","Data":"df71904e1644984fb64b7a9714cedcbaa49bf60df39b8f9aef832199af1a9a98"} Nov 03 13:48:53.577090 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:53.577029 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" podStartSLOduration=2.577017024 podStartE2EDuration="2.577017024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:53.576012688 +0000 UTC m=+2664.889984547" watchObservedRunningTime="2026-11-03 13:48:53.577017024 +0000 UTC m=+2664.890988884" Nov 03 13:48:54.263115 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.263080 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:48:54.264973 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.264947 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 13:48:54.265099 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.264997 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 13:48:54.619181 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.619084 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp"] Nov 03 13:48:54.623407 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.623380 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:48:54.673479 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.673443 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fb4ca0-dockercfg-jpzpp\"" Nov 03 13:48:54.692044 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.692004 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpmr\" (UniqueName: \"kubernetes.io/projected/5f750af2-2ebe-4f41-817a-69d9ac8f867f-kube-api-access-bzpmr\") pod \"payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp\" (UID: \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:48:54.692269 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.692238 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f750af2-2ebe-4f41-817a-69d9ac8f867f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp\" (UID: \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:48:54.720598 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.720566 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fb4ca0\"" Nov 03 13:48:54.725257 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.725222 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp"] Nov 03 13:48:54.777349 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.777297 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft"] Nov 03 13:48:54.784161 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.784131 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:48:54.792968 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.792940 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f750af2-2ebe-4f41-817a-69d9ac8f867f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp\" (UID: \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:48:54.793076 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.793022 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpmr\" (UniqueName: \"kubernetes.io/projected/5f750af2-2ebe-4f41-817a-69d9ac8f867f-kube-api-access-bzpmr\") pod \"payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp\" (UID: \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:48:54.793634 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.793616 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f750af2-2ebe-4f41-817a-69d9ac8f867f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp\" (UID: \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:48:54.830451 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.830416 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft"] Nov 03 13:48:54.864825 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.863833 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpmr\" (UniqueName: \"kubernetes.io/projected/5f750af2-2ebe-4f41-817a-69d9ac8f867f-kube-api-access-bzpmr\") pod \"payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp\" (UID: \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:48:54.894025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.893937 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcszs\" (UniqueName: \"kubernetes.io/projected/e9134245-0e3d-44da-936e-2626e030048f-kube-api-access-vcszs\") pod \"payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft\" (UID: \"e9134245-0e3d-44da-936e-2626e030048f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:48:54.894215 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.894026 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9134245-0e3d-44da-936e-2626e030048f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft\" (UID: \"e9134245-0e3d-44da-936e-2626e030048f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:48:54.955795 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.954995 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:48:54.995507 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.995473 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcszs\" (UniqueName: \"kubernetes.io/projected/e9134245-0e3d-44da-936e-2626e030048f-kube-api-access-vcszs\") pod \"payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft\" (UID: \"e9134245-0e3d-44da-936e-2626e030048f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:48:54.995685 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.995564 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9134245-0e3d-44da-936e-2626e030048f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft\" (UID: \"e9134245-0e3d-44da-936e-2626e030048f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:48:54.996342 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:54.996317 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9134245-0e3d-44da-936e-2626e030048f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft\" (UID: \"e9134245-0e3d-44da-936e-2626e030048f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:48:55.062025 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.061919 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcszs\" (UniqueName: \"kubernetes.io/projected/e9134245-0e3d-44da-936e-2626e030048f-kube-api-access-vcszs\") pod \"payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft\" (UID: \"e9134245-0e3d-44da-936e-2626e030048f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:48:55.097822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.097760 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:48:55.263341 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.263295 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 13:48:55.263541 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.263357 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 13:48:55.328712 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.328683 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp"] Nov 03 13:48:55.330275 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:48:55.330245 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f750af2_2ebe_4f41_817a_69d9ac8f867f.slice/crio-14c474106de3f7ef509f7a52f19163e2fe6247a6079a737cb6a7d9679c1f3497 WatchSource:0}: Error finding container 14c474106de3f7ef509f7a52f19163e2fe6247a6079a737cb6a7d9679c1f3497: Status 404 returned error can't find the container with id 14c474106de3f7ef509f7a52f19163e2fe6247a6079a737cb6a7d9679c1f3497 Nov 03 13:48:55.479558 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.479455 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" event={"ID":"5f750af2-2ebe-4f41-817a-69d9ac8f867f","Type":"ContainerStarted","Data":"9e2a46dea0fb8530997f458bcdee18799acf9b06ec2c6a24b92f8feec4de5b50"} Nov 03 13:48:55.479558 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.479543 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:48:55.479812 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.479572 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" event={"ID":"5f750af2-2ebe-4f41-817a-69d9ac8f867f","Type":"ContainerStarted","Data":"14c474106de3f7ef509f7a52f19163e2fe6247a6079a737cb6a7d9679c1f3497"} Nov 03 13:48:55.491267 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.491244 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft"] Nov 03 13:48:55.492651 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:48:55.492624 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9134245_0e3d_44da_936e_2626e030048f.slice/crio-c42d61e8d9207ac7fc36bc60c36ccd2a98c01bc6b62ee3f4cdd7d3aec2a9b1c0 WatchSource:0}: Error finding container c42d61e8d9207ac7fc36bc60c36ccd2a98c01bc6b62ee3f4cdd7d3aec2a9b1c0: Status 404 returned error can't find the container with id c42d61e8d9207ac7fc36bc60c36ccd2a98c01bc6b62ee3f4cdd7d3aec2a9b1c0 Nov 03 13:48:55.589890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:55.589837 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" podStartSLOduration=1.589822834 podStartE2EDuration="1.589822834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:55.586617415 +0000 UTC m=+2666.900589274" watchObservedRunningTime="2026-11-03 13:48:55.589822834 +0000 UTC m=+2666.903794694" Nov 03 13:48:56.263421 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:56.263377 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 13:48:56.263972 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:56.263451 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 13:48:56.486663 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:56.486629 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" event={"ID":"e9134245-0e3d-44da-936e-2626e030048f","Type":"ContainerStarted","Data":"7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc"} Nov 03 13:48:56.486884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:56.486695 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" event={"ID":"e9134245-0e3d-44da-936e-2626e030048f","Type":"ContainerStarted","Data":"c42d61e8d9207ac7fc36bc60c36ccd2a98c01bc6b62ee3f4cdd7d3aec2a9b1c0"} Nov 03 13:48:56.607742 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:56.607618 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" podStartSLOduration=2.607599934 podStartE2EDuration="2.607599934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:56.606118845 +0000 UTC m=+2667.920090705" watchObservedRunningTime="2026-11-03 13:48:56.607599934 +0000 UTC m=+2667.921571794" Nov 03 13:48:57.263491 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:57.263449 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 13:48:57.263948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:57.263526 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 13:48:57.493307 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:57.493273 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:48:58.263334 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:58.263295 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 13:48:58.263522 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:58.263379 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 13:48:59.263693 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:59.263647 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 03 13:48:59.264230 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:48:59.263708 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 03 13:49:00.268453 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:49:00.268418 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:49:00.268927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:49:00.268879 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:49:00.269589 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:49:00.269566 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:49:01.429677 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:49:01.429643 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 13:49:06.490406 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:49:06.490368 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:49:08.504072 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:49:08.504041 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:49:16.900449 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:49:16.900411 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:49:29.201222 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:49:29.201187 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:49:29.203312 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:49:29.203285 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:49:46.890839 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:49:46.890801 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:50:04.291871 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.291837 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk"] Nov 03 13:50:04.292465 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.292080 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" podUID="fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" containerName="maas-api" containerID="cri-o://3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a" gracePeriod=30 Nov 03 13:50:04.566569 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.566535 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:50:04.653097 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.653059 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-metrics-certs\") pod \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " Nov 03 13:50:04.653097 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.653103 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls\") pod \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " Nov 03 13:50:04.653305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.653134 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-kube-api-access-rmjc5\" (UniqueName: \"kubernetes.io/projected/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-kube-api-access-rmjc5\") pod \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " Nov 03 13:50:04.653305 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.653205 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-openshift-service-ca\") pod \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\" (UID: \"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a\") " Nov 03 13:50:04.653665 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.653628 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-openshift-service-ca" pod "fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" (UID: "fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:04.655178 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.655155 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-metrics-certs" pod "fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" (UID: "fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:04.655317 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.655298 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls" pod "fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" (UID: "fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:50:04.655379 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.655350 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-kube-api-access-rmjc5" pod "fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" (UID: "fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a"). InnerVolumeSpecName "kube-api-access-rmjc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:04.754146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.754106 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:04.754146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.754144 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:04.754146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.754154 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:04.754408 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.754163 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjc5\" (UniqueName: \"kubernetes.io/projected/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a-kube-api-access-rmjc5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:04.895496 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.895464 4541 generic.go:356] "Generic (PLEG): container finished" podID="fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" containerID="3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a" exitCode=0 Nov 03 13:50:04.895660 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.895507 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" Nov 03 13:50:04.895660 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.895529 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" event={"ID":"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a","Type":"ContainerDied","Data":"3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a"} Nov 03 13:50:04.895660 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.895570 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk" event={"ID":"fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a","Type":"ContainerDied","Data":"404e19aa53757fa5aeaf02eb551de64b1cf2712501c29aa13e99270dcdbb8677"} Nov 03 13:50:04.895660 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.895591 4541 scope.go:122] "RemoveContainer" containerID="3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a" Nov 03 13:50:04.905121 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.905098 4541 scope.go:122] "RemoveContainer" containerID="3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a" Nov 03 13:50:04.905375 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:50:04.905354 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a\": container with ID starting with 3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a not found: ID does not exist" containerID="3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a" Nov 03 13:50:04.905492 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:04.905387 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a"} err="failed to get container status \"3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a\": rpc error: code = NotFound desc = could not find container \"3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a\": container with ID starting with 3fd0d64c9ba52c6896f1bb0d8439a63698b9b17e878addd88dd4e2b48dd3666a not found: ID does not exist" Nov 03 13:50:05.052418 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:05.052368 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk"] Nov 03 13:50:05.057820 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:05.057786 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-977f32-75c765d789-wqvnk"] Nov 03 13:50:06.868884 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:06.868853 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" path="/var/lib/kubelet/pods/fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a/volumes" Nov 03 13:50:10.045668 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:10.045586 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8"] Nov 03 13:50:10.092641 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:10.092608 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98f3f387-ggtt8"] Nov 03 13:50:10.869270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:10.869237 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b86734-8ee2-41b5-a17b-6c9924345b89" path="/var/lib/kubelet/pods/34b86734-8ee2-41b5-a17b-6c9924345b89/volumes" Nov 03 13:50:14.527893 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:14.527859 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9"] Nov 03 13:50:16.890900 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:50:16.890856 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:50:19.528176 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.528118 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" podUID="dab28994-6da6-4c95-9a25-de2cc069aa70" containerName="payload-pre-processing" containerID="cri-o://a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66" gracePeriod=25 Nov 03 13:50:19.793560 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.793493 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:50:19.952597 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.952565 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dab28994-6da6-4c95-9a25-de2cc069aa70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab28994-6da6-4c95-9a25-de2cc069aa70-plugins-config-volume\") pod \"dab28994-6da6-4c95-9a25-de2cc069aa70\" (UID: \"dab28994-6da6-4c95-9a25-de2cc069aa70\") " Nov 03 13:50:19.952825 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.952685 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dab28994-6da6-4c95-9a25-de2cc069aa70-kube-api-access-m5k96\" (UniqueName: \"kubernetes.io/projected/dab28994-6da6-4c95-9a25-de2cc069aa70-kube-api-access-m5k96\") pod \"dab28994-6da6-4c95-9a25-de2cc069aa70\" (UID: \"dab28994-6da6-4c95-9a25-de2cc069aa70\") " Nov 03 13:50:19.952990 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.952959 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dab28994-6da6-4c95-9a25-de2cc069aa70-plugins-config-volume" pod "dab28994-6da6-4c95-9a25-de2cc069aa70" (UID: "dab28994-6da6-4c95-9a25-de2cc069aa70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:19.953164 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.953146 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dab28994-6da6-4c95-9a25-de2cc069aa70-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:19.954957 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.954928 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dab28994-6da6-4c95-9a25-de2cc069aa70-kube-api-access-m5k96" pod "dab28994-6da6-4c95-9a25-de2cc069aa70" (UID: "dab28994-6da6-4c95-9a25-de2cc069aa70"). InnerVolumeSpecName "kube-api-access-m5k96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:19.986981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.986948 4541 generic.go:356] "Generic (PLEG): container finished" podID="dab28994-6da6-4c95-9a25-de2cc069aa70" containerID="a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66" exitCode=0 Nov 03 13:50:19.987172 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.987009 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" Nov 03 13:50:19.987172 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.987036 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" event={"ID":"dab28994-6da6-4c95-9a25-de2cc069aa70","Type":"ContainerDied","Data":"a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66"} Nov 03 13:50:19.987172 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.987101 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9" event={"ID":"dab28994-6da6-4c95-9a25-de2cc069aa70","Type":"ContainerDied","Data":"4d497aa9f3400baa0e6d5dc3d0cc5c2b7aacc17f803a5b12649046d96008230a"} Nov 03 13:50:19.987172 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.987131 4541 scope.go:122] "RemoveContainer" containerID="a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66" Nov 03 13:50:19.998530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.998511 4541 scope.go:122] "RemoveContainer" containerID="a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66" Nov 03 13:50:19.998805 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:50:19.998761 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66\": container with ID starting with a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66 not found: ID does not exist" containerID="a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66" Nov 03 13:50:19.998861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:19.998816 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66"} err="failed to get container status \"a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66\": rpc error: code = NotFound desc = could not find container \"a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66\": container with ID starting with a2f7e75b81dbbafbbbb33273c9e5bb7edebf1ba4cbe1f15ba9b39d80e2865f66 not found: ID does not exist" Nov 03 13:50:20.054127 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:20.054045 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5k96\" (UniqueName: \"kubernetes.io/projected/dab28994-6da6-4c95-9a25-de2cc069aa70-kube-api-access-m5k96\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:20.142938 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:20.142900 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9"] Nov 03 13:50:20.147320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:20.147281 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-10c688-55896dfd4c-xqns9"] Nov 03 13:50:20.868573 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:50:20.868540 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dab28994-6da6-4c95-9a25-de2cc069aa70" path="/var/lib/kubelet/pods/dab28994-6da6-4c95-9a25-de2cc069aa70/volumes" Nov 03 13:50:46.890822 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:50:46.890764 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:51:07.170825 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.170791 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj"] Nov 03 13:51:07.171529 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.171506 4541 cpu_manager.go:414] "containerMap: removing container" podUID="dab28994-6da6-4c95-9a25-de2cc069aa70" containerName="payload-pre-processing" Nov 03 13:51:07.171603 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.171532 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dab28994-6da6-4c95-9a25-de2cc069aa70" containerName="payload-pre-processing" Nov 03 13:51:07.171603 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.171555 4541 cpu_manager.go:414] "containerMap: removing container" podUID="fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" containerName="maas-api" Nov 03 13:51:07.171603 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.171562 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" containerName="maas-api" Nov 03 13:51:07.171765 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.171710 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="dab28994-6da6-4c95-9a25-de2cc069aa70" containerName="payload-pre-processing" Nov 03 13:51:07.171765 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.171720 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd1aba03-7251-4bfe-a5d1-e2d1f8ef023a" containerName="maas-api" Nov 03 13:51:07.175360 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.175341 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:07.227338 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.227306 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-d0231d\"" Nov 03 13:51:07.274444 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.274408 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-d0231d-dockercfg-b7pr2\"" Nov 03 13:51:07.277988 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.277954 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj"] Nov 03 13:51:07.300878 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.300750 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzv5k\" (UniqueName: \"kubernetes.io/projected/a01491e5-7f57-4419-9186-c2966e8cc22f-kube-api-access-bzv5k\") pod \"payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj\" (UID: \"a01491e5-7f57-4419-9186-c2966e8cc22f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:07.300878 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.300833 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01491e5-7f57-4419-9186-c2966e8cc22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj\" (UID: \"a01491e5-7f57-4419-9186-c2966e8cc22f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:07.370581 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.370543 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7"] Nov 03 13:51:07.374880 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.374855 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:07.401698 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.401664 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256691b8-991d-4105-9b10-40dbcc6dcc2c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7\" (UID: \"256691b8-991d-4105-9b10-40dbcc6dcc2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:07.401832 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.401709 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk5s8\" (UniqueName: \"kubernetes.io/projected/256691b8-991d-4105-9b10-40dbcc6dcc2c-kube-api-access-sk5s8\") pod \"payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7\" (UID: \"256691b8-991d-4105-9b10-40dbcc6dcc2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:07.401907 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.401886 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzv5k\" (UniqueName: \"kubernetes.io/projected/a01491e5-7f57-4419-9186-c2966e8cc22f-kube-api-access-bzv5k\") pod \"payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj\" (UID: \"a01491e5-7f57-4419-9186-c2966e8cc22f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:07.401963 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.401925 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01491e5-7f57-4419-9186-c2966e8cc22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj\" (UID: \"a01491e5-7f57-4419-9186-c2966e8cc22f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:07.402520 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.402498 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01491e5-7f57-4419-9186-c2966e8cc22f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj\" (UID: \"a01491e5-7f57-4419-9186-c2966e8cc22f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:07.429091 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.429020 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7"] Nov 03 13:51:07.456439 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.456411 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzv5k\" (UniqueName: \"kubernetes.io/projected/a01491e5-7f57-4419-9186-c2966e8cc22f-kube-api-access-bzv5k\") pod \"payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj\" (UID: \"a01491e5-7f57-4419-9186-c2966e8cc22f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:07.492440 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.492412 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:07.503225 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.503191 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256691b8-991d-4105-9b10-40dbcc6dcc2c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7\" (UID: \"256691b8-991d-4105-9b10-40dbcc6dcc2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:07.503373 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.503235 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk5s8\" (UniqueName: \"kubernetes.io/projected/256691b8-991d-4105-9b10-40dbcc6dcc2c-kube-api-access-sk5s8\") pod \"payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7\" (UID: \"256691b8-991d-4105-9b10-40dbcc6dcc2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:07.503892 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.503870 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256691b8-991d-4105-9b10-40dbcc6dcc2c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7\" (UID: \"256691b8-991d-4105-9b10-40dbcc6dcc2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:07.561839 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.558185 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk5s8\" (UniqueName: \"kubernetes.io/projected/256691b8-991d-4105-9b10-40dbcc6dcc2c-kube-api-access-sk5s8\") pod \"payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7\" (UID: \"256691b8-991d-4105-9b10-40dbcc6dcc2c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:07.688208 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.688108 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:07.867164 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:07.867132 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj"] Nov 03 13:51:07.867981 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:51:07.867938 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01491e5_7f57_4419_9186_c2966e8cc22f.slice/crio-3b0ad39c79a0996c53d7b02452fb89910894a27cf1c3dc7529f013f99ef53c30 WatchSource:0}: Error finding container 3b0ad39c79a0996c53d7b02452fb89910894a27cf1c3dc7529f013f99ef53c30: Status 404 returned error can't find the container with id 3b0ad39c79a0996c53d7b02452fb89910894a27cf1c3dc7529f013f99ef53c30 Nov 03 13:51:08.056133 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:08.056095 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7"] Nov 03 13:51:08.059818 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:51:08.059788 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256691b8_991d_4105_9b10_40dbcc6dcc2c.slice/crio-08a18980967c5c9a1ea43a7effb28ce97a9364dda00dcdb2bfea0050a97c9bb3 WatchSource:0}: Error finding container 08a18980967c5c9a1ea43a7effb28ce97a9364dda00dcdb2bfea0050a97c9bb3: Status 404 returned error can't find the container with id 08a18980967c5c9a1ea43a7effb28ce97a9364dda00dcdb2bfea0050a97c9bb3 Nov 03 13:51:08.273734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:08.273638 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" event={"ID":"a01491e5-7f57-4419-9186-c2966e8cc22f","Type":"ContainerStarted","Data":"1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b"} Nov 03 13:51:08.273734 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:08.273696 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" event={"ID":"a01491e5-7f57-4419-9186-c2966e8cc22f","Type":"ContainerStarted","Data":"3b0ad39c79a0996c53d7b02452fb89910894a27cf1c3dc7529f013f99ef53c30"} Nov 03 13:51:08.274257 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:08.273797 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:08.275364 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:08.275334 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" event={"ID":"256691b8-991d-4105-9b10-40dbcc6dcc2c","Type":"ContainerStarted","Data":"6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95"} Nov 03 13:51:08.275485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:08.275381 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" event={"ID":"256691b8-991d-4105-9b10-40dbcc6dcc2c","Type":"ContainerStarted","Data":"08a18980967c5c9a1ea43a7effb28ce97a9364dda00dcdb2bfea0050a97c9bb3"} Nov 03 13:51:08.275485 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:08.275412 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:08.384943 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:08.384886 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" podStartSLOduration=1.384872326 podStartE2EDuration="1.384872326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:08.38260947 +0000 UTC m=+2799.696581342" watchObservedRunningTime="2026-11-03 13:51:08.384872326 +0000 UTC m=+2799.698844186" Nov 03 13:51:08.493976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:08.493923 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" podStartSLOduration=1.4939089939999999 podStartE2EDuration="1.493908994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:08.492568257 +0000 UTC m=+2799.806540117" watchObservedRunningTime="2026-11-03 13:51:08.493908994 +0000 UTC m=+2799.807880853" Nov 03 13:51:13.361438 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.361401 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv"] Nov 03 13:51:13.361948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.361600 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" podUID="0b44cba8-006c-4ba8-959e-ff72144a7dec" containerName="maas-api" containerID="cri-o://29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074" gracePeriod=30 Nov 03 13:51:13.624638 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.624613 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:51:13.684459 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.684427 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls\") pod \"0b44cba8-006c-4ba8-959e-ff72144a7dec\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " Nov 03 13:51:13.684634 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.684482 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b44cba8-006c-4ba8-959e-ff72144a7dec-kube-api-access-mqsqh\" (UniqueName: \"kubernetes.io/projected/0b44cba8-006c-4ba8-959e-ff72144a7dec-kube-api-access-mqsqh\") pod \"0b44cba8-006c-4ba8-959e-ff72144a7dec\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " Nov 03 13:51:13.684634 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.684610 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-metrics-certs\") pod \"0b44cba8-006c-4ba8-959e-ff72144a7dec\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " Nov 03 13:51:13.684714 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.684644 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b44cba8-006c-4ba8-959e-ff72144a7dec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b44cba8-006c-4ba8-959e-ff72144a7dec-openshift-service-ca\") pod \"0b44cba8-006c-4ba8-959e-ff72144a7dec\" (UID: \"0b44cba8-006c-4ba8-959e-ff72144a7dec\") " Nov 03 13:51:13.685098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.685072 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b44cba8-006c-4ba8-959e-ff72144a7dec-openshift-service-ca" pod "0b44cba8-006c-4ba8-959e-ff72144a7dec" (UID: "0b44cba8-006c-4ba8-959e-ff72144a7dec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:13.686675 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.686654 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-metrics-certs" pod "0b44cba8-006c-4ba8-959e-ff72144a7dec" (UID: "0b44cba8-006c-4ba8-959e-ff72144a7dec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:51:13.686741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.686672 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b44cba8-006c-4ba8-959e-ff72144a7dec-kube-api-access-mqsqh" pod "0b44cba8-006c-4ba8-959e-ff72144a7dec" (UID: "0b44cba8-006c-4ba8-959e-ff72144a7dec"). InnerVolumeSpecName "kube-api-access-mqsqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:13.686741 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.686690 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls" pod "0b44cba8-006c-4ba8-959e-ff72144a7dec" (UID: "0b44cba8-006c-4ba8-959e-ff72144a7dec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:51:13.786416 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.786378 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:13.786416 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.786415 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqsqh\" (UniqueName: \"kubernetes.io/projected/0b44cba8-006c-4ba8-959e-ff72144a7dec-kube-api-access-mqsqh\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:13.786633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.786427 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b44cba8-006c-4ba8-959e-ff72144a7dec-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:13.786633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:13.786436 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b44cba8-006c-4ba8-959e-ff72144a7dec-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:14.315668 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.315629 4541 generic.go:356] "Generic (PLEG): container finished" podID="0b44cba8-006c-4ba8-959e-ff72144a7dec" containerID="29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074" exitCode=0 Nov 03 13:51:14.315856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.315671 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" Nov 03 13:51:14.315856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.315691 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" event={"ID":"0b44cba8-006c-4ba8-959e-ff72144a7dec","Type":"ContainerDied","Data":"29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074"} Nov 03 13:51:14.315856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.315758 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv" event={"ID":"0b44cba8-006c-4ba8-959e-ff72144a7dec","Type":"ContainerDied","Data":"59e7d43b8503eccd99342e3214e1217806b97a4829751b8122c47ceb50ca7ccb"} Nov 03 13:51:14.315856 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.315810 4541 scope.go:122] "RemoveContainer" containerID="29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074" Nov 03 13:51:14.325585 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.325565 4541 scope.go:122] "RemoveContainer" containerID="29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074" Nov 03 13:51:14.325894 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:51:14.325872 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074\": container with ID starting with 29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074 not found: ID does not exist" containerID="29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074" Nov 03 13:51:14.325939 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.325904 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074"} err="failed to get container status \"29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074\": rpc error: code = NotFound desc = could not find container \"29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074\": container with ID starting with 29978a0675e7f7af9b3c8d05a188aad1278d984d523924a786b87dc729185074 not found: ID does not exist" Nov 03 13:51:14.470987 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.470948 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv"] Nov 03 13:51:14.518063 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.518021 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-10c688-674c97d594-vd2tv"] Nov 03 13:51:14.868877 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:14.868843 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b44cba8-006c-4ba8-959e-ff72144a7dec" path="/var/lib/kubelet/pods/0b44cba8-006c-4ba8-959e-ff72144a7dec/volumes" Nov 03 13:51:16.889815 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:51:16.889759 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:51:19.284855 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:19.284826 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 13:51:19.285279 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:19.284894 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 13:51:29.451105 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:29.451076 4541 scope.go:122] "RemoveContainer" containerID="0a365ae9062a7ee4670fc6d9d96c83743a39bdd2d233deec85060cb3db20a8d4" Nov 03 13:51:41.987555 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:41.987469 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm"] Nov 03 13:51:41.988112 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:41.987660 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" podUID="276a435a-e9f8-4867-884a-e2c45184e26e" containerName="maas-api" containerID="cri-o://c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4" gracePeriod=30 Nov 03 13:51:42.299748 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.299722 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:51:42.392723 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.392687 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/276a435a-e9f8-4867-884a-e2c45184e26e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276a435a-e9f8-4867-884a-e2c45184e26e-openshift-service-ca\") pod \"276a435a-e9f8-4867-884a-e2c45184e26e\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " Nov 03 13:51:42.392928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.392738 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-metrics-certs\") pod \"276a435a-e9f8-4867-884a-e2c45184e26e\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " Nov 03 13:51:42.392928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.392839 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276a435a-e9f8-4867-884a-e2c45184e26e-kube-api-access-xj8rp\" (UniqueName: \"kubernetes.io/projected/276a435a-e9f8-4867-884a-e2c45184e26e-kube-api-access-xj8rp\") pod \"276a435a-e9f8-4867-884a-e2c45184e26e\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " Nov 03 13:51:42.392928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.392866 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls\") pod \"276a435a-e9f8-4867-884a-e2c45184e26e\" (UID: \"276a435a-e9f8-4867-884a-e2c45184e26e\") " Nov 03 13:51:42.393170 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.393144 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276a435a-e9f8-4867-884a-e2c45184e26e-openshift-service-ca" pod "276a435a-e9f8-4867-884a-e2c45184e26e" (UID: "276a435a-e9f8-4867-884a-e2c45184e26e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:42.395404 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.395360 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls" pod "276a435a-e9f8-4867-884a-e2c45184e26e" (UID: "276a435a-e9f8-4867-884a-e2c45184e26e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:51:42.395525 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.395424 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276a435a-e9f8-4867-884a-e2c45184e26e-kube-api-access-xj8rp" pod "276a435a-e9f8-4867-884a-e2c45184e26e" (UID: "276a435a-e9f8-4867-884a-e2c45184e26e"). InnerVolumeSpecName "kube-api-access-xj8rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:42.395979 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.395950 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-metrics-certs" pod "276a435a-e9f8-4867-884a-e2c45184e26e" (UID: "276a435a-e9f8-4867-884a-e2c45184e26e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:51:42.477968 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.477931 4541 generic.go:356] "Generic (PLEG): container finished" podID="276a435a-e9f8-4867-884a-e2c45184e26e" containerID="c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4" exitCode=0 Nov 03 13:51:42.478165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.477986 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" Nov 03 13:51:42.478165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.478017 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" event={"ID":"276a435a-e9f8-4867-884a-e2c45184e26e","Type":"ContainerDied","Data":"c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4"} Nov 03 13:51:42.478165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.478075 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm" event={"ID":"276a435a-e9f8-4867-884a-e2c45184e26e","Type":"ContainerDied","Data":"7267f8e7191877184f3d097708b70065c63814d18c5377791bb243324cf4f481"} Nov 03 13:51:42.478165 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.478096 4541 scope.go:122] "RemoveContainer" containerID="c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4" Nov 03 13:51:42.488846 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.488819 4541 scope.go:122] "RemoveContainer" containerID="c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4" Nov 03 13:51:42.489182 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:51:42.489155 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4\": container with ID starting with c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4 not found: ID does not exist" containerID="c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4" Nov 03 13:51:42.489266 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.489189 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4"} err="failed to get container status \"c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4\": rpc error: code = NotFound desc = could not find container \"c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4\": container with ID starting with c85cb7619e460e3b3be6e8196da57f5f1b19e6b85bd8e56a44d7bb55165343c4 not found: ID does not exist" Nov 03 13:51:42.493759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.493735 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/276a435a-e9f8-4867-884a-e2c45184e26e-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:42.493864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.493761 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:42.493864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.493796 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj8rp\" (UniqueName: \"kubernetes.io/projected/276a435a-e9f8-4867-884a-e2c45184e26e-kube-api-access-xj8rp\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:42.493864 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.493807 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/276a435a-e9f8-4867-884a-e2c45184e26e-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:42.633819 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.633681 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm"] Nov 03 13:51:42.637372 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.637344 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d83c93-5b558f96dc-jjjcm"] Nov 03 13:51:42.868380 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:42.868337 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276a435a-e9f8-4867-884a-e2c45184e26e" path="/var/lib/kubelet/pods/276a435a-e9f8-4867-884a-e2c45184e26e/volumes" Nov 03 13:51:46.890502 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:51:46.890462 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:51:49.127102 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.127068 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992"] Nov 03 13:51:49.127687 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.127671 4541 cpu_manager.go:414] "containerMap: removing container" podUID="0b44cba8-006c-4ba8-959e-ff72144a7dec" containerName="maas-api" Nov 03 13:51:49.127784 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.127689 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b44cba8-006c-4ba8-959e-ff72144a7dec" containerName="maas-api" Nov 03 13:51:49.127784 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.127710 4541 cpu_manager.go:414] "containerMap: removing container" podUID="276a435a-e9f8-4867-884a-e2c45184e26e" containerName="maas-api" Nov 03 13:51:49.127784 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.127715 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276a435a-e9f8-4867-884a-e2c45184e26e" containerName="maas-api" Nov 03 13:51:49.127923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.127845 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b44cba8-006c-4ba8-959e-ff72144a7dec" containerName="maas-api" Nov 03 13:51:49.127923 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.127860 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="276a435a-e9f8-4867-884a-e2c45184e26e" containerName="maas-api" Nov 03 13:51:49.132567 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.132543 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.168193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.168158 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.168193 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.168201 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.168407 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.168341 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8sjg\" (UniqueName: \"kubernetes.io/projected/4e3e2e04-551b-4094-b4d4-6f35b41069ef-kube-api-access-b8sjg\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.168486 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.168471 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3e2e04-551b-4094-b4d4-6f35b41069ef-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.182017 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.181991 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-d341af-serving-cert\"" Nov 03 13:51:49.232073 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.232036 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992"] Nov 03 13:51:49.269600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.269565 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3e2e04-551b-4094-b4d4-6f35b41069ef-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.269600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.269628 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.269600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.269666 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.269600 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.269745 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8sjg\" (UniqueName: \"kubernetes.io/projected/4e3e2e04-551b-4094-b4d4-6f35b41069ef-kube-api-access-b8sjg\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.269600 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:51:49.269879 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-serving-cert: secret "maas-api-e2e-shared-b-w2-d341af-serving-cert" not found Nov 03 13:51:49.269600 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:51:49.269977 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls podName:4e3e2e04-551b-4094-b4d4-6f35b41069ef nodeName:}" failed. No retries permitted until 2026-11-03 13:51:49.769954759 +0000 UTC m=+2841.083926615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls") pod "maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" (UID: "4e3e2e04-551b-4094-b4d4-6f35b41069ef") : secret "maas-api-e2e-shared-b-w2-d341af-serving-cert" not found Nov 03 13:51:49.270703 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.270465 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3e2e04-551b-4094-b4d4-6f35b41069ef-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.274074 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.274049 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.325078 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.325042 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8sjg\" (UniqueName: \"kubernetes.io/projected/4e3e2e04-551b-4094-b4d4-6f35b41069ef-kube-api-access-b8sjg\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.775927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.775883 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:49.778291 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:49.778270 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:50.046858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:50.046735 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:50.415209 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:50.415179 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992"] Nov 03 13:51:50.416411 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:51:50.416382 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e3e2e04_551b_4094_b4d4_6f35b41069ef.slice/crio-10c08befdfac795f574b4164f47f13552c1d5cf18fc16e725cd6365a80d312bd WatchSource:0}: Error finding container 10c08befdfac795f574b4164f47f13552c1d5cf18fc16e725cd6365a80d312bd: Status 404 returned error can't find the container with id 10c08befdfac795f574b4164f47f13552c1d5cf18fc16e725cd6365a80d312bd Nov 03 13:51:50.418047 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:50.418025 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:51:50.531860 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:50.531819 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" event={"ID":"4e3e2e04-551b-4094-b4d4-6f35b41069ef","Type":"ContainerStarted","Data":"10c08befdfac795f574b4164f47f13552c1d5cf18fc16e725cd6365a80d312bd"} Nov 03 13:51:51.538930 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:51.538893 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" event={"ID":"4e3e2e04-551b-4094-b4d4-6f35b41069ef","Type":"ContainerStarted","Data":"050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751"} Nov 03 13:51:51.539407 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:51.538963 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:51.652071 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:51.652021 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" podStartSLOduration=1.696426454 podStartE2EDuration="2.652008152s" totalImagesPullingTime="955.581698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:51.64932949 +0000 UTC m=+2842.963301351" watchObservedRunningTime="2026-11-03 13:51:51.652008152 +0000 UTC m=+2842.965980012" Nov 03 13:51:57.549829 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:57.549795 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 13:51:58.403472 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.403439 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh"] Nov 03 13:51:58.407555 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.407536 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.471706 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.471662 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185d31d-49f8-4a83-a724-5b2219356e3d-istiod-ca-cert\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.471904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.471739 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npdhc\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-kube-api-access-npdhc\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.471904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.471791 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-socket\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.471904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.471816 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-podinfo\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.471904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.471864 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-data\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.471904 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.471897 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-envoy\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.472126 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.471992 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-credential-socket\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.472126 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.472069 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-certs\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.472126 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.472116 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-token\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.483055 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.483023 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh"] Nov 03 13:51:58.509502 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.509463 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-5ae63d-openshift-default-dockercfg-cvwnc\"" Nov 03 13:51:58.573054 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573014 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-credential-socket\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573093 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-certs\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573124 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-token\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573156 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185d31d-49f8-4a83-a724-5b2219356e3d-istiod-ca-cert\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573191 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npdhc\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-kube-api-access-npdhc\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573216 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-socket\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573231 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-podinfo\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573383 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-data\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573429 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-envoy\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573471 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-credential-socket\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.573524 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573499 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-certs\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.574054 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573533 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-socket\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.574054 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.573719 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-data\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.574218 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.574195 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185d31d-49f8-4a83-a724-5b2219356e3d-istiod-ca-cert\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.575658 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.575635 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-envoy\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.575883 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.575864 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-podinfo\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.628314 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.628285 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-token\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.628459 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.628432 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npdhc\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-kube-api-access-npdhc\") pod \"e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:58.721338 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:58.721246 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:59.140244 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:59.140151 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh"] Nov 03 13:51:59.175073 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:51:59.175041 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode185d31d_49f8_4a83_a724_5b2219356e3d.slice/crio-8f2b89794d506975452e0f3e89de09a9de2c0e2703bd7df245ba00d9d35b8c60 WatchSource:0}: Error finding container 8f2b89794d506975452e0f3e89de09a9de2c0e2703bd7df245ba00d9d35b8c60: Status 404 returned error can't find the container with id 8f2b89794d506975452e0f3e89de09a9de2c0e2703bd7df245ba00d9d35b8c60 Nov 03 13:51:59.177791 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:59.177740 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:51:59.177891 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:59.177822 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:51:59.177891 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:59.177854 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:51:59.588788 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:59.588738 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" event={"ID":"e185d31d-49f8-4a83-a724-5b2219356e3d","Type":"ContainerStarted","Data":"1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e"} Nov 03 13:51:59.589225 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:59.588822 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" event={"ID":"e185d31d-49f8-4a83-a724-5b2219356e3d","Type":"ContainerStarted","Data":"8f2b89794d506975452e0f3e89de09a9de2c0e2703bd7df245ba00d9d35b8c60"} Nov 03 13:51:59.721860 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:59.721831 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:51:59.796134 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:51:59.796076 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" podStartSLOduration=1.796064251 podStartE2EDuration="1.796064251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:59.74812305 +0000 UTC m=+2851.062094910" watchObservedRunningTime="2026-11-03 13:51:59.796064251 +0000 UTC m=+2851.110036110" Nov 03 13:52:00.728107 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:00.728080 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:52:01.600048 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:01.600013 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:52:01.601127 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:01.601106 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:52:06.315199 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.315143 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8"] Nov 03 13:52:06.319560 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.319539 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:06.369669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.369636 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-5ae63d-dockercfg-t4fnw\"" Nov 03 13:52:06.373265 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.373237 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8"] Nov 03 13:52:06.417097 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.417066 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-5ae63d\"" Nov 03 13:52:06.469638 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.469589 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9n6w\" (UniqueName: \"kubernetes.io/projected/9890570b-67ea-47e3-b607-c1456210c7a6-kube-api-access-z9n6w\") pod \"payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8\" (UID: \"9890570b-67ea-47e3-b607-c1456210c7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:06.469848 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.469700 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9890570b-67ea-47e3-b607-c1456210c7a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8\" (UID: \"9890570b-67ea-47e3-b607-c1456210c7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:06.474001 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.473975 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx"] Nov 03 13:52:06.478370 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.478345 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:06.525677 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.525642 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx"] Nov 03 13:52:06.571021 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.570928 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9n6w\" (UniqueName: \"kubernetes.io/projected/9890570b-67ea-47e3-b607-c1456210c7a6-kube-api-access-z9n6w\") pod \"payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8\" (UID: \"9890570b-67ea-47e3-b607-c1456210c7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:06.571194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.571039 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9890570b-67ea-47e3-b607-c1456210c7a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8\" (UID: \"9890570b-67ea-47e3-b607-c1456210c7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:06.571194 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.571139 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e8525a-86b2-440e-8215-3431a27a6a54-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx\" (UID: \"16e8525a-86b2-440e-8215-3431a27a6a54\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:06.571299 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.571190 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56rx\" (UniqueName: \"kubernetes.io/projected/16e8525a-86b2-440e-8215-3431a27a6a54-kube-api-access-w56rx\") pod \"payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx\" (UID: \"16e8525a-86b2-440e-8215-3431a27a6a54\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:06.571763 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.571741 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9890570b-67ea-47e3-b607-c1456210c7a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8\" (UID: \"9890570b-67ea-47e3-b607-c1456210c7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:06.625525 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.625485 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9n6w\" (UniqueName: \"kubernetes.io/projected/9890570b-67ea-47e3-b607-c1456210c7a6-kube-api-access-z9n6w\") pod \"payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8\" (UID: \"9890570b-67ea-47e3-b607-c1456210c7a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:06.633786 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.633747 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:06.672662 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.672620 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e8525a-86b2-440e-8215-3431a27a6a54-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx\" (UID: \"16e8525a-86b2-440e-8215-3431a27a6a54\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:06.672847 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.672703 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56rx\" (UniqueName: \"kubernetes.io/projected/16e8525a-86b2-440e-8215-3431a27a6a54-kube-api-access-w56rx\") pod \"payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx\" (UID: \"16e8525a-86b2-440e-8215-3431a27a6a54\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:06.673394 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.673370 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e8525a-86b2-440e-8215-3431a27a6a54-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx\" (UID: \"16e8525a-86b2-440e-8215-3431a27a6a54\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:06.727443 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.727408 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56rx\" (UniqueName: \"kubernetes.io/projected/16e8525a-86b2-440e-8215-3431a27a6a54-kube-api-access-w56rx\") pod \"payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx\" (UID: \"16e8525a-86b2-440e-8215-3431a27a6a54\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:06.805379 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:06.805345 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:07.002154 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:52:07.002121 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9890570b_67ea_47e3_b607_c1456210c7a6.slice/crio-35d0a01b489ce82a3eb6bc3266b6df2ee1221c9527dd916133ee75559d14f596 WatchSource:0}: Error finding container 35d0a01b489ce82a3eb6bc3266b6df2ee1221c9527dd916133ee75559d14f596: Status 404 returned error can't find the container with id 35d0a01b489ce82a3eb6bc3266b6df2ee1221c9527dd916133ee75559d14f596 Nov 03 13:52:07.002709 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.002684 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8"] Nov 03 13:52:07.174605 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.174575 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx"] Nov 03 13:52:07.175037 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:52:07.175008 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e8525a_86b2_440e_8215_3431a27a6a54.slice/crio-6febd9cf3d6839f80df71ee54687b9541332b80fc43b5bc030195a2ab17eb40c WatchSource:0}: Error finding container 6febd9cf3d6839f80df71ee54687b9541332b80fc43b5bc030195a2ab17eb40c: Status 404 returned error can't find the container with id 6febd9cf3d6839f80df71ee54687b9541332b80fc43b5bc030195a2ab17eb40c Nov 03 13:52:07.634888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.634757 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" event={"ID":"16e8525a-86b2-440e-8215-3431a27a6a54","Type":"ContainerStarted","Data":"776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638"} Nov 03 13:52:07.634888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.634854 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:07.634888 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.634878 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" event={"ID":"16e8525a-86b2-440e-8215-3431a27a6a54","Type":"ContainerStarted","Data":"6febd9cf3d6839f80df71ee54687b9541332b80fc43b5bc030195a2ab17eb40c"} Nov 03 13:52:07.636439 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.636410 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" event={"ID":"9890570b-67ea-47e3-b607-c1456210c7a6","Type":"ContainerStarted","Data":"59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10"} Nov 03 13:52:07.636558 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.636463 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:07.636558 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.636480 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" event={"ID":"9890570b-67ea-47e3-b607-c1456210c7a6","Type":"ContainerStarted","Data":"35d0a01b489ce82a3eb6bc3266b6df2ee1221c9527dd916133ee75559d14f596"} Nov 03 13:52:07.746436 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.746381 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" podStartSLOduration=1.746366231 podStartE2EDuration="1.746366231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:52:07.744273814 +0000 UTC m=+2859.058245675" watchObservedRunningTime="2026-11-03 13:52:07.746366231 +0000 UTC m=+2859.060338116" Nov 03 13:52:07.852835 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:07.852744 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" podStartSLOduration=1.852728254 podStartE2EDuration="1.852728254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:52:07.851234096 +0000 UTC m=+2859.165205955" watchObservedRunningTime="2026-11-03 13:52:07.852728254 +0000 UTC m=+2859.166700117" Nov 03 13:52:09.389679 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:09.389641 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft"] Nov 03 13:52:09.497372 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:09.497334 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp"] Nov 03 13:52:14.390175 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.390130 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" podUID="e9134245-0e3d-44da-936e-2626e030048f" containerName="payload-processing" containerID="cri-o://7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc" gracePeriod=25 Nov 03 13:52:14.497670 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.497615 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" podUID="5f750af2-2ebe-4f41-817a-69d9ac8f867f" containerName="payload-pre-processing" containerID="cri-o://9e2a46dea0fb8530997f458bcdee18799acf9b06ec2c6a24b92f8feec4de5b50" gracePeriod=25 Nov 03 13:52:14.667951 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.667919 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:52:14.685712 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.685676 4541 generic.go:356] "Generic (PLEG): container finished" podID="e9134245-0e3d-44da-936e-2626e030048f" containerID="7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc" exitCode=0 Nov 03 13:52:14.685890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.685754 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" Nov 03 13:52:14.685890 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.685755 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" event={"ID":"e9134245-0e3d-44da-936e-2626e030048f","Type":"ContainerDied","Data":"7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc"} Nov 03 13:52:14.686003 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.685934 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft" event={"ID":"e9134245-0e3d-44da-936e-2626e030048f","Type":"ContainerDied","Data":"c42d61e8d9207ac7fc36bc60c36ccd2a98c01bc6b62ee3f4cdd7d3aec2a9b1c0"} Nov 03 13:52:14.686003 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.685960 4541 scope.go:122] "RemoveContainer" containerID="7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc" Nov 03 13:52:14.688106 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.688080 4541 generic.go:356] "Generic (PLEG): container finished" podID="5f750af2-2ebe-4f41-817a-69d9ac8f867f" containerID="9e2a46dea0fb8530997f458bcdee18799acf9b06ec2c6a24b92f8feec4de5b50" exitCode=0 Nov 03 13:52:14.688279 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.688255 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" event={"ID":"5f750af2-2ebe-4f41-817a-69d9ac8f867f","Type":"ContainerDied","Data":"9e2a46dea0fb8530997f458bcdee18799acf9b06ec2c6a24b92f8feec4de5b50"} Nov 03 13:52:14.698320 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.698291 4541 scope.go:122] "RemoveContainer" containerID="7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc" Nov 03 13:52:14.698589 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:52:14.698570 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc\": container with ID starting with 7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc not found: ID does not exist" containerID="7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc" Nov 03 13:52:14.698639 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.698599 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc"} err="failed to get container status \"7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc\": rpc error: code = NotFound desc = could not find container \"7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc\": container with ID starting with 7922b5a85f74d577ae5c03ee136df9d1a66d399ae8e6f90dcd21e294758ba4cc not found: ID does not exist" Nov 03 13:52:14.765196 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.765169 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:52:14.765956 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.765937 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9134245-0e3d-44da-936e-2626e030048f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9134245-0e3d-44da-936e-2626e030048f-plugins-config-volume\") pod \"e9134245-0e3d-44da-936e-2626e030048f\" (UID: \"e9134245-0e3d-44da-936e-2626e030048f\") " Nov 03 13:52:14.766098 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.766084 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9134245-0e3d-44da-936e-2626e030048f-kube-api-access-vcszs\" (UniqueName: \"kubernetes.io/projected/e9134245-0e3d-44da-936e-2626e030048f-kube-api-access-vcszs\") pod \"e9134245-0e3d-44da-936e-2626e030048f\" (UID: \"e9134245-0e3d-44da-936e-2626e030048f\") " Nov 03 13:52:14.766279 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.766258 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9134245-0e3d-44da-936e-2626e030048f-plugins-config-volume" pod "e9134245-0e3d-44da-936e-2626e030048f" (UID: "e9134245-0e3d-44da-936e-2626e030048f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:52:14.766447 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.766434 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9134245-0e3d-44da-936e-2626e030048f-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:14.768043 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.768022 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9134245-0e3d-44da-936e-2626e030048f-kube-api-access-vcszs" pod "e9134245-0e3d-44da-936e-2626e030048f" (UID: "e9134245-0e3d-44da-936e-2626e030048f"). InnerVolumeSpecName "kube-api-access-vcszs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:52:14.867142 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.867109 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f750af2-2ebe-4f41-817a-69d9ac8f867f-kube-api-access-bzpmr\" (UniqueName: \"kubernetes.io/projected/5f750af2-2ebe-4f41-817a-69d9ac8f867f-kube-api-access-bzpmr\") pod \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\" (UID: \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\") " Nov 03 13:52:14.867339 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.867204 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f750af2-2ebe-4f41-817a-69d9ac8f867f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f750af2-2ebe-4f41-817a-69d9ac8f867f-plugins-config-volume\") pod \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\" (UID: \"5f750af2-2ebe-4f41-817a-69d9ac8f867f\") " Nov 03 13:52:14.867545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.867529 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcszs\" (UniqueName: \"kubernetes.io/projected/e9134245-0e3d-44da-936e-2626e030048f-kube-api-access-vcszs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:14.867598 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.867525 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f750af2-2ebe-4f41-817a-69d9ac8f867f-plugins-config-volume" pod "5f750af2-2ebe-4f41-817a-69d9ac8f867f" (UID: "5f750af2-2ebe-4f41-817a-69d9ac8f867f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:52:14.869289 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.869262 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f750af2-2ebe-4f41-817a-69d9ac8f867f-kube-api-access-bzpmr" pod "5f750af2-2ebe-4f41-817a-69d9ac8f867f" (UID: "5f750af2-2ebe-4f41-817a-69d9ac8f867f"). InnerVolumeSpecName "kube-api-access-bzpmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:52:14.968683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.968546 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzpmr\" (UniqueName: \"kubernetes.io/projected/5f750af2-2ebe-4f41-817a-69d9ac8f867f-kube-api-access-bzpmr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:14.968683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:14.968636 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f750af2-2ebe-4f41-817a-69d9ac8f867f-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:15.095255 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:15.095211 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft"] Nov 03 13:52:15.099646 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:15.099615 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fb4ca0-796bd68545-wn6ft"] Nov 03 13:52:15.696822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:15.696714 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" Nov 03 13:52:15.696822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:15.696743 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp" event={"ID":"5f750af2-2ebe-4f41-817a-69d9ac8f867f","Type":"ContainerDied","Data":"14c474106de3f7ef509f7a52f19163e2fe6247a6079a737cb6a7d9679c1f3497"} Nov 03 13:52:15.696822 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:15.696820 4541 scope.go:122] "RemoveContainer" containerID="9e2a46dea0fb8530997f458bcdee18799acf9b06ec2c6a24b92f8feec4de5b50" Nov 03 13:52:15.853139 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:15.853094 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp"] Nov 03 13:52:15.899759 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:15.899725 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fb4ca0-56657bb59f-chcdp"] Nov 03 13:52:16.869260 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:16.869228 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f750af2-2ebe-4f41-817a-69d9ac8f867f" path="/var/lib/kubelet/pods/5f750af2-2ebe-4f41-817a-69d9ac8f867f/volumes" Nov 03 13:52:16.869650 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:16.869628 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9134245-0e3d-44da-936e-2626e030048f" path="/var/lib/kubelet/pods/e9134245-0e3d-44da-936e-2626e030048f/volumes" Nov 03 13:52:16.889932 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:52:16.889896 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:52:18.644974 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:18.644944 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 13:52:18.645337 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:52:18.645011 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 13:52:46.893807 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:52:46.893751 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:53:00.581087 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.581048 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt"] Nov 03 13:53:00.581627 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.581611 4541 cpu_manager.go:414] "containerMap: removing container" podUID="5f750af2-2ebe-4f41-817a-69d9ac8f867f" containerName="payload-pre-processing" Nov 03 13:53:00.581675 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.581627 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f750af2-2ebe-4f41-817a-69d9ac8f867f" containerName="payload-pre-processing" Nov 03 13:53:00.581675 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.581662 4541 cpu_manager.go:414] "containerMap: removing container" podUID="e9134245-0e3d-44da-936e-2626e030048f" containerName="payload-processing" Nov 03 13:53:00.581675 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.581667 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9134245-0e3d-44da-936e-2626e030048f" containerName="payload-processing" Nov 03 13:53:00.581813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.581753 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9134245-0e3d-44da-936e-2626e030048f" containerName="payload-processing" Nov 03 13:53:00.581813 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.581766 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f750af2-2ebe-4f41-817a-69d9ac8f867f" containerName="payload-pre-processing" Nov 03 13:53:00.585314 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.585294 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:00.684441 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.683350 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-367381\"" Nov 03 13:53:00.684441 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.683375 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-367381-dockercfg-h7c57\"" Nov 03 13:53:00.686696 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.686666 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt"] Nov 03 13:53:00.715649 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.715615 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdff9c10-2c0e-4384-a569-099f2d0800ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt\" (UID: \"fdff9c10-2c0e-4384-a569-099f2d0800ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:00.715874 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.715850 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mbxf\" (UniqueName: \"kubernetes.io/projected/fdff9c10-2c0e-4384-a569-099f2d0800ba-kube-api-access-2mbxf\") pod \"payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt\" (UID: \"fdff9c10-2c0e-4384-a569-099f2d0800ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:00.726946 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.726916 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl"] Nov 03 13:53:00.731077 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.731057 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:00.785339 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.785305 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl"] Nov 03 13:53:00.817263 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.817224 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4cf945-e803-4428-9287-633fb3653273-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl\" (UID: \"1f4cf945-e803-4428-9287-633fb3653273\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:00.817423 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.817384 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8lg6\" (UniqueName: \"kubernetes.io/projected/1f4cf945-e803-4428-9287-633fb3653273-kube-api-access-x8lg6\") pod \"payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl\" (UID: \"1f4cf945-e803-4428-9287-633fb3653273\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:00.817561 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.817535 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mbxf\" (UniqueName: \"kubernetes.io/projected/fdff9c10-2c0e-4384-a569-099f2d0800ba-kube-api-access-2mbxf\") pod \"payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt\" (UID: \"fdff9c10-2c0e-4384-a569-099f2d0800ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:00.817671 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.817652 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdff9c10-2c0e-4384-a569-099f2d0800ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt\" (UID: \"fdff9c10-2c0e-4384-a569-099f2d0800ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:00.818606 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.818575 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdff9c10-2c0e-4384-a569-099f2d0800ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt\" (UID: \"fdff9c10-2c0e-4384-a569-099f2d0800ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:00.872270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.872196 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mbxf\" (UniqueName: \"kubernetes.io/projected/fdff9c10-2c0e-4384-a569-099f2d0800ba-kube-api-access-2mbxf\") pod \"payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt\" (UID: \"fdff9c10-2c0e-4384-a569-099f2d0800ba\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:00.898944 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.898913 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:00.919104 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.919072 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4cf945-e803-4428-9287-633fb3653273-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl\" (UID: \"1f4cf945-e803-4428-9287-633fb3653273\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:00.919227 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.919181 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8lg6\" (UniqueName: \"kubernetes.io/projected/1f4cf945-e803-4428-9287-633fb3653273-kube-api-access-x8lg6\") pod \"payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl\" (UID: \"1f4cf945-e803-4428-9287-633fb3653273\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:00.919753 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.919730 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4cf945-e803-4428-9287-633fb3653273-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl\" (UID: \"1f4cf945-e803-4428-9287-633fb3653273\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:00.974953 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:00.974920 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8lg6\" (UniqueName: \"kubernetes.io/projected/1f4cf945-e803-4428-9287-633fb3653273-kube-api-access-x8lg6\") pod \"payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl\" (UID: \"1f4cf945-e803-4428-9287-633fb3653273\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:01.045731 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:01.045694 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:01.269365 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:01.269337 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt"] Nov 03 13:53:01.271011 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:53:01.270980 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdff9c10_2c0e_4384_a569_099f2d0800ba.slice/crio-72a16093df3309f9b8eb72576b44882b9efcb1bad57b3784aab222caee0bb51b WatchSource:0}: Error finding container 72a16093df3309f9b8eb72576b44882b9efcb1bad57b3784aab222caee0bb51b: Status 404 returned error can't find the container with id 72a16093df3309f9b8eb72576b44882b9efcb1bad57b3784aab222caee0bb51b Nov 03 13:53:01.413575 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:01.413546 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl"] Nov 03 13:53:01.415100 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:53:01.415071 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4cf945_e803_4428_9287_633fb3653273.slice/crio-dc886c86395ab21e9a465371bbf3870ed404b8c7ad6141de0351e124ddfdf3a7 WatchSource:0}: Error finding container dc886c86395ab21e9a465371bbf3870ed404b8c7ad6141de0351e124ddfdf3a7: Status 404 returned error can't find the container with id dc886c86395ab21e9a465371bbf3870ed404b8c7ad6141de0351e124ddfdf3a7 Nov 03 13:53:01.983367 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:01.983334 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" event={"ID":"fdff9c10-2c0e-4384-a569-099f2d0800ba","Type":"ContainerStarted","Data":"40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43"} Nov 03 13:53:01.983854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:01.983387 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" event={"ID":"fdff9c10-2c0e-4384-a569-099f2d0800ba","Type":"ContainerStarted","Data":"72a16093df3309f9b8eb72576b44882b9efcb1bad57b3784aab222caee0bb51b"} Nov 03 13:53:01.983854 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:01.983546 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:01.985093 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:01.985066 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" event={"ID":"1f4cf945-e803-4428-9287-633fb3653273","Type":"ContainerStarted","Data":"f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b"} Nov 03 13:53:01.985204 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:01.985116 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" event={"ID":"1f4cf945-e803-4428-9287-633fb3653273","Type":"ContainerStarted","Data":"dc886c86395ab21e9a465371bbf3870ed404b8c7ad6141de0351e124ddfdf3a7"} Nov 03 13:53:01.985204 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:01.985147 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:02.093147 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:02.093078 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" podStartSLOduration=2.093058462 podStartE2EDuration="2.093058462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:02.090217301 +0000 UTC m=+2913.404189160" watchObservedRunningTime="2026-11-03 13:53:02.093058462 +0000 UTC m=+2913.407030322" Nov 03 13:53:02.197993 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:02.197932 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" podStartSLOduration=2.197918265 podStartE2EDuration="2.197918265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:02.19725295 +0000 UTC m=+2913.511224823" watchObservedRunningTime="2026-11-03 13:53:02.197918265 +0000 UTC m=+2913.511890124" Nov 03 13:53:09.030832 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:09.030792 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9"] Nov 03 13:53:09.032167 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:09.032125 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" containerID="cri-o://2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95" gracePeriod=30 Nov 03 13:53:12.993792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:12.993694 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 13:53:12.993792 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:12.993761 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 13:53:13.961367 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:13.961324 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:53:13.961532 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:13.961405 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:53:14.296908 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.296883 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:53:14.377322 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377283 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-envoy\") pod \"9e510d28-293c-449b-a23e-9e32fe24895b\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " Nov 03 13:53:14.377489 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377341 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-credential-socket\") pod \"9e510d28-293c-449b-a23e-9e32fe24895b\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " Nov 03 13:53:14.377489 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377380 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e510d28-293c-449b-a23e-9e32fe24895b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e510d28-293c-449b-a23e-9e32fe24895b-istiod-ca-cert\") pod \"9e510d28-293c-449b-a23e-9e32fe24895b\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " Nov 03 13:53:14.377489 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377437 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-data\") pod \"9e510d28-293c-449b-a23e-9e32fe24895b\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " Nov 03 13:53:14.377489 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377474 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-socket\") pod \"9e510d28-293c-449b-a23e-9e32fe24895b\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " Nov 03 13:53:14.377717 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377576 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-kube-api-access-sz279\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-kube-api-access-sz279\") pod \"9e510d28-293c-449b-a23e-9e32fe24895b\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " Nov 03 13:53:14.377717 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377621 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-istio-token\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-istio-token\") pod \"9e510d28-293c-449b-a23e-9e32fe24895b\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " Nov 03 13:53:14.377717 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377644 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-credential-socket" pod "9e510d28-293c-449b-a23e-9e32fe24895b" (UID: "9e510d28-293c-449b-a23e-9e32fe24895b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.377921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377719 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-certs\") pod \"9e510d28-293c-449b-a23e-9e32fe24895b\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " Nov 03 13:53:14.377921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377804 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e510d28-293c-449b-a23e-9e32fe24895b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e510d28-293c-449b-a23e-9e32fe24895b-istio-podinfo\") pod \"9e510d28-293c-449b-a23e-9e32fe24895b\" (UID: \"9e510d28-293c-449b-a23e-9e32fe24895b\") " Nov 03 13:53:14.378011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377736 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-data" pod "9e510d28-293c-449b-a23e-9e32fe24895b" (UID: "9e510d28-293c-449b-a23e-9e32fe24895b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.378011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377854 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e510d28-293c-449b-a23e-9e32fe24895b-istiod-ca-cert" pod "9e510d28-293c-449b-a23e-9e32fe24895b" (UID: "9e510d28-293c-449b-a23e-9e32fe24895b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:53:14.378011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.377855 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-socket" pod "9e510d28-293c-449b-a23e-9e32fe24895b" (UID: "9e510d28-293c-449b-a23e-9e32fe24895b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.378153 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.378023 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-certs" pod "9e510d28-293c-449b-a23e-9e32fe24895b" (UID: "9e510d28-293c-449b-a23e-9e32fe24895b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.378309 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.378292 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.378378 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.378314 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e510d28-293c-449b-a23e-9e32fe24895b-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.378378 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.378327 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.378378 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.378340 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.378378 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.378354 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.379849 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.379823 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-kube-api-access-sz279" pod "9e510d28-293c-449b-a23e-9e32fe24895b" (UID: "9e510d28-293c-449b-a23e-9e32fe24895b"). InnerVolumeSpecName "kube-api-access-sz279". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.380065 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.380041 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-envoy" pod "9e510d28-293c-449b-a23e-9e32fe24895b" (UID: "9e510d28-293c-449b-a23e-9e32fe24895b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.380137 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.380109 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-istio-token" pod "9e510d28-293c-449b-a23e-9e32fe24895b" (UID: "9e510d28-293c-449b-a23e-9e32fe24895b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.380177 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.380130 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e510d28-293c-449b-a23e-9e32fe24895b-istio-podinfo" pod "9e510d28-293c-449b-a23e-9e32fe24895b" (UID: "9e510d28-293c-449b-a23e-9e32fe24895b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:53:14.479390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.479357 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e510d28-293c-449b-a23e-9e32fe24895b-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.479390 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.479389 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e510d28-293c-449b-a23e-9e32fe24895b-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.479593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.479404 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz279\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-kube-api-access-sz279\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.479593 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:14.479417 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e510d28-293c-449b-a23e-9e32fe24895b-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:15.065084 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:15.065048 4541 generic.go:356] "Generic (PLEG): container finished" podID="9e510d28-293c-449b-a23e-9e32fe24895b" containerID="2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95" exitCode=0 Nov 03 13:53:15.065253 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:15.065102 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" Nov 03 13:53:15.065253 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:15.065134 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" event={"ID":"9e510d28-293c-449b-a23e-9e32fe24895b","Type":"ContainerDied","Data":"2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95"} Nov 03 13:53:15.065253 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:15.065199 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9" event={"ID":"9e510d28-293c-449b-a23e-9e32fe24895b","Type":"ContainerDied","Data":"1c8dd83d084cfc8b2f4565b3f9b3ac3a78fd50dad33eee7d3b84a427d86a24e7"} Nov 03 13:53:15.065253 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:15.065230 4541 scope.go:122] "RemoveContainer" containerID="2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95" Nov 03 13:53:15.074635 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:15.074616 4541 scope.go:122] "RemoveContainer" containerID="2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95" Nov 03 13:53:15.074943 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:53:15.074922 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95\": container with ID starting with 2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95 not found: ID does not exist" containerID="2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95" Nov 03 13:53:15.075012 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:15.074953 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95"} err="failed to get container status \"2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95\": rpc error: code = NotFound desc = could not find container \"2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95\": container with ID starting with 2ef3ba886cc9e3286a538f17530c847868dcbe84a1efe04bc3b616855986ef95 not found: ID does not exist" Nov 03 13:53:15.269456 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:15.269389 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9"] Nov 03 13:53:15.272781 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:15.272742 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lxnj9"] Nov 03 13:53:16.868921 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:16.868888 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" path="/var/lib/kubelet/pods/9e510d28-293c-449b-a23e-9e32fe24895b/volumes" Nov 03 13:53:16.890503 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:53:16.890468 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:53:43.244814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:43.244708 4541 generic.go:356] "Generic (PLEG): container finished" podID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerID="1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e" exitCode=0 Nov 03 13:53:43.244814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:43.244793 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" event={"ID":"e185d31d-49f8-4a83-a724-5b2219356e3d","Type":"ContainerDied","Data":"1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e"} Nov 03 13:53:43.245285 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:43.245209 4541 scope.go:122] "RemoveContainer" containerID="1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e" Nov 03 13:53:43.246830 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:43.246802 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:53:43.246913 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:43.246872 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:53:43.246913 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:43.246908 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 13:53:44.267058 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:44.267025 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" event={"ID":"e185d31d-49f8-4a83-a724-5b2219356e3d","Type":"ContainerStarted","Data":"e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680"} Nov 03 13:53:44.721980 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:44.721936 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:53:44.723141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:44.723115 4541 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 13:53:44.723251 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:44.723162 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 13:53:45.721998 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:45.721963 4541 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 13:53:45.722413 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:45.722020 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 13:53:46.600674 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:46.600628 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:53:46.722257 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:46.722217 4541 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 13:53:46.722743 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:46.722297 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 13:53:46.890678 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:53:46.890597 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:53:47.722329 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:47.722294 4541 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 13:53:47.722726 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:47.722349 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 13:53:48.721954 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:48.721914 4541 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body= Nov 03 13:53:48.722146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:48.721982 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" Nov 03 13:53:49.727285 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:49.727252 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:53:49.728279 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:53:49.728260 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 13:54:07.548180 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.548128 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps"] Nov 03 13:54:07.548803 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.548764 4541 cpu_manager.go:414] "containerMap: removing container" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" Nov 03 13:54:07.548803 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.548801 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" Nov 03 13:54:07.549010 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.548934 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e510d28-293c-449b-a23e-9e32fe24895b" containerName="istio-proxy" Nov 03 13:54:07.551296 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.551274 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.629311 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.629275 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-home\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.629491 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.629343 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-model-cache\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.629491 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.629422 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-dshm\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.629491 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.629465 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tmp-dir\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.629491 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.629490 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w88xx\" (UniqueName: \"kubernetes.io/projected/2bb00b65-8ce6-43ec-939e-ecdded328aaf-kube-api-access-w88xx\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.629732 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.629653 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tls-certs\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.650705 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.650671 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5faf5d\"/\"default-dockercfg-qs4b7\"" Nov 03 13:54:07.650705 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.650671 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5faf5d\"/\"openshift-service-ca.crt\"" Nov 03 13:54:07.650705 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.650679 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5faf5d\"/\"kube-root-ca.crt\"" Nov 03 13:54:07.650981 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.650835 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5faf5d\"/\"auth-test-model-5faf5d-kserve-self-signed-certs\"" Nov 03 13:54:07.653833 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.653813 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps"] Nov 03 13:54:07.730812 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.730765 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-model-cache\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.730812 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.730819 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-dshm\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.731074 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.730842 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tmp-dir\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.731074 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.730859 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w88xx\" (UniqueName: \"kubernetes.io/projected/2bb00b65-8ce6-43ec-939e-ecdded328aaf-kube-api-access-w88xx\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.731074 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.730968 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tls-certs\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.731074 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.731038 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-home\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.731290 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.731251 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-model-cache\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.731332 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.731282 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tmp-dir\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.731429 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.731402 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-home\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.733244 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.733225 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-dshm\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.733364 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.733346 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tls-certs\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.785568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.785535 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w88xx\" (UniqueName: \"kubernetes.io/projected/2bb00b65-8ce6-43ec-939e-ecdded328aaf-kube-api-access-w88xx\") pod \"auth-test-model-5faf5d-kserve-7468d547d5-mzjps\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:07.861669 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:07.861572 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:08.232233 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:08.232200 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps"] Nov 03 13:54:08.233214 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:54:08.233177 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bb00b65_8ce6_43ec_939e_ecdded328aaf.slice/crio-373fd194cd7513ceeeff47eeef34d35e3db893cb0010fe87f76a369037fb7660 WatchSource:0}: Error finding container 373fd194cd7513ceeeff47eeef34d35e3db893cb0010fe87f76a369037fb7660: Status 404 returned error can't find the container with id 373fd194cd7513ceeeff47eeef34d35e3db893cb0010fe87f76a369037fb7660 Nov 03 13:54:08.413989 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:08.413954 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" event={"ID":"2bb00b65-8ce6-43ec-939e-ecdded328aaf","Type":"ContainerStarted","Data":"8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406"} Nov 03 13:54:08.414166 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:08.413998 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" event={"ID":"2bb00b65-8ce6-43ec-939e-ecdded328aaf","Type":"ContainerStarted","Data":"373fd194cd7513ceeeff47eeef34d35e3db893cb0010fe87f76a369037fb7660"} Nov 03 13:54:08.525381 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:08.525278 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" podStartSLOduration=1.525263549 podStartE2EDuration="1.525263549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:54:08.52217395 +0000 UTC m=+2979.836145809" watchObservedRunningTime="2026-11-03 13:54:08.525263549 +0000 UTC m=+2979.839235409" Nov 03 13:54:08.869356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:08.869275 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:09.462928 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:09.462890 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2"] Nov 03 13:54:09.463129 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:09.463107 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" containerID="cri-o://3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408" gracePeriod=30 Nov 03 13:54:14.741647 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.741586 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:54:14.815895 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.815860 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istiod-ca-cert\") pod \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " Nov 03 13:54:14.816091 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.815909 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-data\") pod \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " Nov 03 13:54:14.816091 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816045 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-kube-api-access-6nkjq\") pod \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " Nov 03 13:54:14.816091 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816083 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-certs\") pod \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " Nov 03 13:54:14.816274 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816143 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-credential-socket\") pod \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " Nov 03 13:54:14.816274 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816182 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-envoy\") pod \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " Nov 03 13:54:14.816274 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816220 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-data" pod "a8d505a0-416c-4607-a9f4-58c4a3916f1b" (UID: "a8d505a0-416c-4607-a9f4-58c4a3916f1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.816274 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816237 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-podinfo\") pod \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " Nov 03 13:54:14.816274 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816261 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istiod-ca-cert" pod "a8d505a0-416c-4607-a9f4-58c4a3916f1b" (UID: "a8d505a0-416c-4607-a9f4-58c4a3916f1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:54:14.816536 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816278 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-socket\") pod \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " Nov 03 13:54:14.816536 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816411 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-certs" pod "a8d505a0-416c-4607-a9f4-58c4a3916f1b" (UID: "a8d505a0-416c-4607-a9f4-58c4a3916f1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.816536 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816430 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-credential-socket" pod "a8d505a0-416c-4607-a9f4-58c4a3916f1b" (UID: "a8d505a0-416c-4607-a9f4-58c4a3916f1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.816536 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816498 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-token\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-token\") pod \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\" (UID: \"a8d505a0-416c-4607-a9f4-58c4a3916f1b\") " Nov 03 13:54:14.816536 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816503 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-socket" pod "a8d505a0-416c-4607-a9f4-58c4a3916f1b" (UID: "a8d505a0-416c-4607-a9f4-58c4a3916f1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.816978 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816952 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.817135 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816980 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.817135 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.816996 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.817135 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.817013 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.817135 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.817029 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.818447 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.818421 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-envoy" pod "a8d505a0-416c-4607-a9f4-58c4a3916f1b" (UID: "a8d505a0-416c-4607-a9f4-58c4a3916f1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.818447 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.818437 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-podinfo" pod "a8d505a0-416c-4607-a9f4-58c4a3916f1b" (UID: "a8d505a0-416c-4607-a9f4-58c4a3916f1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:54:14.818568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.818456 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-kube-api-access-6nkjq" pod "a8d505a0-416c-4607-a9f4-58c4a3916f1b" (UID: "a8d505a0-416c-4607-a9f4-58c4a3916f1b"). InnerVolumeSpecName "kube-api-access-6nkjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:54:14.818568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.818489 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-token" pod "a8d505a0-416c-4607-a9f4-58c4a3916f1b" (UID: "a8d505a0-416c-4607-a9f4-58c4a3916f1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:54:14.918011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.917974 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkjq\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-kube-api-access-6nkjq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.918011 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.918012 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.918216 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.918026 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.918216 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:14.918038 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8d505a0-416c-4607-a9f4-58c4a3916f1b-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:15.457288 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:15.457256 4541 generic.go:356] "Generic (PLEG): container finished" podID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerID="3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408" exitCode=0 Nov 03 13:54:15.457479 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:15.457306 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" Nov 03 13:54:15.457479 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:15.457332 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" event={"ID":"a8d505a0-416c-4607-a9f4-58c4a3916f1b","Type":"ContainerDied","Data":"3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408"} Nov 03 13:54:15.457479 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:15.457382 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2" event={"ID":"a8d505a0-416c-4607-a9f4-58c4a3916f1b","Type":"ContainerDied","Data":"df71904e1644984fb64b7a9714cedcbaa49bf60df39b8f9aef832199af1a9a98"} Nov 03 13:54:15.457479 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:15.457403 4541 scope.go:122] "RemoveContainer" containerID="3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408" Nov 03 13:54:15.467341 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:15.467324 4541 scope.go:122] "RemoveContainer" containerID="3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408" Nov 03 13:54:15.467637 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:54:15.467612 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408\": container with ID starting with 3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408 not found: ID does not exist" containerID="3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408" Nov 03 13:54:15.467748 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:15.467644 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408"} err="failed to get container status \"3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408\": rpc error: code = NotFound desc = could not find container \"3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408\": container with ID starting with 3ba2b79eeb119ffb36ec72ee4645e8bc2e3fb93da274618a6495fa184f661408 not found: ID does not exist" Nov 03 13:54:15.660762 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:15.660721 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2"] Nov 03 13:54:15.708358 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:15.708252 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-zn8c2"] Nov 03 13:54:16.869655 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:16.869625 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" path="/var/lib/kubelet/pods/a8d505a0-416c-4607-a9f4-58c4a3916f1b/volumes" Nov 03 13:54:16.890131 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:54:16.890093 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:54:17.861697 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:17.861658 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:17.874138 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:17.874108 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:18.486203 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:18.486170 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:54:29.238356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:29.238324 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:54:29.241307 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:54:29.241286 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:54:46.890748 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:54:46.890710 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:55:16.891287 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:55:16.891250 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:55:20.535462 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.535421 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm"] Nov 03 13:55:20.536009 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.535996 4541 cpu_manager.go:414] "containerMap: removing container" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" Nov 03 13:55:20.536064 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.536010 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" Nov 03 13:55:20.536131 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.536122 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d505a0-416c-4607-a9f4-58c4a3916f1b" containerName="istio-proxy" Nov 03 13:55:20.539680 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.539660 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.577331 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.577297 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tmp-dir\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.577484 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.577344 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-model-cache\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.577484 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.577463 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4x76\" (UniqueName: \"kubernetes.io/projected/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-kube-api-access-h4x76\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.577597 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.577578 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-dshm\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.577633 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.577609 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tls-certs\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.577747 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.577728 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-home\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.637816 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.637757 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-d0231d\"/\"openshift-service-ca.crt\"" Nov 03 13:55:20.637816 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.637805 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-d0231d\"/\"kube-root-ca.crt\"" Nov 03 13:55:20.638052 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.637877 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-d0231d\"/\"default-dockercfg-r6nmt\"" Nov 03 13:55:20.638052 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.637892 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-d0231d\"/\"auth-test-model-d0231d-kserve-self-signed-certs\"" Nov 03 13:55:20.640638 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.640614 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm"] Nov 03 13:55:20.679314 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.679284 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tmp-dir\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.679514 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.679330 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-model-cache\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.679514 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.679396 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4x76\" (UniqueName: \"kubernetes.io/projected/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-kube-api-access-h4x76\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.679514 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.679492 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-dshm\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.679683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.679514 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tls-certs\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.679683 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.679644 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-home\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.679812 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.679757 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tmp-dir\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.679906 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.679879 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-model-cache\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.680050 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.680029 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-home\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.682559 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.682533 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-dshm\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.682818 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.682795 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tls-certs\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.733841 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.733802 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4x76\" (UniqueName: \"kubernetes.io/projected/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-kube-api-access-h4x76\") pod \"auth-test-model-d0231d-kserve-dcd957cc-gr4cm\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:20.852310 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:20.852212 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:21.223297 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:21.223255 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm"] Nov 03 13:55:21.224179 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:55:21.224151 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42a7cbb7_e710_4b4a_9fd5_d0ce099d7a03.slice/crio-7cee4e21bb90ee5f72eccb014bd021a4e13d8964e9134cc01116c53006385f5c WatchSource:0}: Error finding container 7cee4e21bb90ee5f72eccb014bd021a4e13d8964e9134cc01116c53006385f5c: Status 404 returned error can't find the container with id 7cee4e21bb90ee5f72eccb014bd021a4e13d8964e9134cc01116c53006385f5c Nov 03 13:55:21.848220 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:21.848176 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" event={"ID":"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03","Type":"ContainerStarted","Data":"24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de"} Nov 03 13:55:21.848585 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:21.848229 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" event={"ID":"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03","Type":"ContainerStarted","Data":"7cee4e21bb90ee5f72eccb014bd021a4e13d8964e9134cc01116c53006385f5c"} Nov 03 13:55:21.852338 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:21.852319 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:21.957927 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:21.957824 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" podStartSLOduration=1.957811024 podStartE2EDuration="1.957811024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:55:21.956660657 +0000 UTC m=+3053.270632519" watchObservedRunningTime="2026-11-03 13:55:21.957811024 +0000 UTC m=+3053.271782884" Nov 03 13:55:30.852798 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:30.852736 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:30.869511 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:30.869485 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:30.915493 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:30.915459 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 13:55:44.617205 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.617167 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv"] Nov 03 13:55:44.621303 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.621281 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.672591 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.672551 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-efbdec-serving-cert\"" Nov 03 13:55:44.723190 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.723154 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv"] Nov 03 13:55:44.760236 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.760192 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.760441 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.760255 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.760441 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.760342 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc8b7\" (UniqueName: \"kubernetes.io/projected/4f83df78-c34c-4c09-b36a-f819e54ddebb-kube-api-access-tc8b7\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.760655 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.760627 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83df78-c34c-4c09-b36a-f819e54ddebb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.864716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.864331 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83df78-c34c-4c09-b36a-f819e54ddebb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.864716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.864422 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.864716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.864451 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.864716 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.864476 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc8b7\" (UniqueName: \"kubernetes.io/projected/4f83df78-c34c-4c09-b36a-f819e54ddebb-kube-api-access-tc8b7\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.866863 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.865616 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83df78-c34c-4c09-b36a-f819e54ddebb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.866863 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:55:44.865712 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-serving-cert: secret "maas-api-e2e-shared-b-w4-efbdec-serving-cert" not found Nov 03 13:55:44.866863 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:55:44.865794 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls podName:4f83df78-c34c-4c09-b36a-f819e54ddebb nodeName:}" failed. No retries permitted until 2026-11-03 13:55:45.365759357 +0000 UTC m=+3076.679731212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls") pod "maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" (UID: "4f83df78-c34c-4c09-b36a-f819e54ddebb") : secret "maas-api-e2e-shared-b-w4-efbdec-serving-cert" not found Nov 03 13:55:44.868846 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.868300 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:44.919120 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:44.919080 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc8b7\" (UniqueName: \"kubernetes.io/projected/4f83df78-c34c-4c09-b36a-f819e54ddebb-kube-api-access-tc8b7\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:45.370673 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:45.370631 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:45.373153 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:45.373127 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:45.537509 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:45.537466 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:45.907452 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:45.907419 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv"] Nov 03 13:55:45.908636 ip-10-0-132-174 kubenswrapper[4541]: W1103 13:55:45.908604 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f83df78_c34c_4c09_b36a_f819e54ddebb.slice/crio-10486ba109a4b6d9c3cb3b11927057cc8279dd06e917d340a8df9b8340ff5d01 WatchSource:0}: Error finding container 10486ba109a4b6d9c3cb3b11927057cc8279dd06e917d340a8df9b8340ff5d01: Status 404 returned error can't find the container with id 10486ba109a4b6d9c3cb3b11927057cc8279dd06e917d340a8df9b8340ff5d01 Nov 03 13:55:45.992401 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:45.992365 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" event={"ID":"4f83df78-c34c-4c09-b36a-f819e54ddebb","Type":"ContainerStarted","Data":"10486ba109a4b6d9c3cb3b11927057cc8279dd06e917d340a8df9b8340ff5d01"} Nov 03 13:55:46.893743 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:55:46.893709 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:55:47.001789 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:47.001737 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" event={"ID":"4f83df78-c34c-4c09-b36a-f819e54ddebb","Type":"ContainerStarted","Data":"a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa"} Nov 03 13:55:47.126977 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:47.126869 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" podStartSLOduration=2.216959047 podStartE2EDuration="3.126854323s" totalImagesPullingTime="909.895276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:55:47.125477069 +0000 UTC m=+3078.439448940" watchObservedRunningTime="2026-11-03 13:55:47.126854323 +0000 UTC m=+3078.440826184" Nov 03 13:55:48.008336 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:48.008298 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:55:54.020091 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:55:54.020061 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 13:56:16.889551 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:56:16.889468 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:56:46.890458 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:56:46.890423 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:57:16.892084 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:57:16.892042 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:57:41.150287 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:41.150203 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j"] Nov 03 13:57:41.150756 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:41.150452 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" containerID="cri-o://19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89" gracePeriod=30 Nov 03 13:57:44.960704 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:44.960667 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:57:44.961092 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:44.960734 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:57:46.406849 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.406823 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:57:46.518356 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518271 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-socket\") pod \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " Nov 03 13:57:46.518530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518384 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-kube-api-access-99spt\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-kube-api-access-99spt\") pod \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " Nov 03 13:57:46.518530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518427 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-certs\") pod \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " Nov 03 13:57:46.518530 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518466 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istiod-ca-cert\") pod \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " Nov 03 13:57:46.518661 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518532 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-token\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-token\") pod \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " Nov 03 13:57:46.518661 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518593 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-podinfo\") pod \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " Nov 03 13:57:46.518661 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518640 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-credential-socket\") pod \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " Nov 03 13:57:46.518861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518695 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-envoy\") pod \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " Nov 03 13:57:46.518861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518684 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-socket" pod "ef2fafe5-bc7b-4291-943c-c501430b7cf4" (UID: "ef2fafe5-bc7b-4291-943c-c501430b7cf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.518861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518739 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-certs" pod "ef2fafe5-bc7b-4291-943c-c501430b7cf4" (UID: "ef2fafe5-bc7b-4291-943c-c501430b7cf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.518861 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518760 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-data\") pod \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\" (UID: \"ef2fafe5-bc7b-4291-943c-c501430b7cf4\") " Nov 03 13:57:46.519084 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.518994 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-credential-socket" pod "ef2fafe5-bc7b-4291-943c-c501430b7cf4" (UID: "ef2fafe5-bc7b-4291-943c-c501430b7cf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.519084 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.519004 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istiod-ca-cert" pod "ef2fafe5-bc7b-4291-943c-c501430b7cf4" (UID: "ef2fafe5-bc7b-4291-943c-c501430b7cf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:57:46.519254 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.519226 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-data" pod "ef2fafe5-bc7b-4291-943c-c501430b7cf4" (UID: "ef2fafe5-bc7b-4291-943c-c501430b7cf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.519397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.519358 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.519397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.519378 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.519397 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.519392 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.519578 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.519407 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.519578 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.519421 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.521141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.521109 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-podinfo" pod "ef2fafe5-bc7b-4291-943c-c501430b7cf4" (UID: "ef2fafe5-bc7b-4291-943c-c501430b7cf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:57:46.521141 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.521113 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-envoy" pod "ef2fafe5-bc7b-4291-943c-c501430b7cf4" (UID: "ef2fafe5-bc7b-4291-943c-c501430b7cf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.521277 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.521186 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-token" pod "ef2fafe5-bc7b-4291-943c-c501430b7cf4" (UID: "ef2fafe5-bc7b-4291-943c-c501430b7cf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:46.521438 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.521417 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-kube-api-access-99spt" pod "ef2fafe5-bc7b-4291-943c-c501430b7cf4" (UID: "ef2fafe5-bc7b-4291-943c-c501430b7cf4"). InnerVolumeSpecName "kube-api-access-99spt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:46.620917 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.620861 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.620917 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.620904 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.620917 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.620915 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99spt\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-kube-api-access-99spt\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.620917 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.620926 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef2fafe5-bc7b-4291-943c-c501430b7cf4-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.720239 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.720203 4541 generic.go:356] "Generic (PLEG): container finished" podID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerID="19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89" exitCode=0 Nov 03 13:57:46.720450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.720257 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" Nov 03 13:57:46.720450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.720303 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" event={"ID":"ef2fafe5-bc7b-4291-943c-c501430b7cf4","Type":"ContainerDied","Data":"19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89"} Nov 03 13:57:46.720450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.720367 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j" event={"ID":"ef2fafe5-bc7b-4291-943c-c501430b7cf4","Type":"ContainerDied","Data":"7f5388b570d9d2fdfc0e708c2f1ba554d7bcc6fbe1ccbb58537d9319af3359e4"} Nov 03 13:57:46.720450 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.720404 4541 scope.go:122] "RemoveContainer" containerID="19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89" Nov 03 13:57:46.730873 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.730832 4541 scope.go:122] "RemoveContainer" containerID="19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89" Nov 03 13:57:46.731128 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:57:46.731110 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89\": container with ID starting with 19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89 not found: ID does not exist" containerID="19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89" Nov 03 13:57:46.731169 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.731137 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89"} err="failed to get container status \"19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89\": rpc error: code = NotFound desc = could not find container \"19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89\": container with ID starting with 19d6cd7362539aca84c0b5a415d5cd809940e9e01d58066652988c69081ede89 not found: ID does not exist" Nov 03 13:57:46.876995 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.876916 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j"] Nov 03 13:57:46.883535 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:46.883500 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fzm9j"] Nov 03 13:57:46.891091 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:57:46.891052 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:57:48.868568 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:57:48.868533 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" path="/var/lib/kubelet/pods/ef2fafe5-bc7b-4291-943c-c501430b7cf4/volumes" Nov 03 13:58:16.893355 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:58:16.893317 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:58:46.890375 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:58:46.890335 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:59:11.844206 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:11.844116 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m"] Nov 03 13:59:11.844749 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:11.844354 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" containerID="cri-o://6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7" gracePeriod=30 Nov 03 13:59:14.799814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:14.799745 4541 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:59:14.800229 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:14.799848 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:59:16.893279 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:59:16.893232 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 13:59:17.138250 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.138217 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:59:17.166649 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.166607 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-certs\") pod \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " Nov 03 13:59:17.166858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.166672 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-token\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-token\") pod \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " Nov 03 13:59:17.166858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.166731 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-envoy\") pod \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " Nov 03 13:59:17.166858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.166805 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istiod-ca-cert\") pod \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " Nov 03 13:59:17.167055 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.166902 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-certs" pod "b6fcb72e-f1e8-4e88-ac56-fde05b65847c" (UID: "b6fcb72e-f1e8-4e88-ac56-fde05b65847c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:17.167055 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.166928 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-podinfo\") pod \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " Nov 03 13:59:17.167055 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.166992 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-data\") pod \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " Nov 03 13:59:17.167055 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.167051 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-socket\") pod \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " Nov 03 13:59:17.167273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.167079 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-kube-api-access-xftvk\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-kube-api-access-xftvk\") pod \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " Nov 03 13:59:17.167273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.167132 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-credential-socket\") pod \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\" (UID: \"b6fcb72e-f1e8-4e88-ac56-fde05b65847c\") " Nov 03 13:59:17.167273 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.167240 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istiod-ca-cert" pod "b6fcb72e-f1e8-4e88-ac56-fde05b65847c" (UID: "b6fcb72e-f1e8-4e88-ac56-fde05b65847c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:59:17.167498 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.167475 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-data" pod "b6fcb72e-f1e8-4e88-ac56-fde05b65847c" (UID: "b6fcb72e-f1e8-4e88-ac56-fde05b65847c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:17.167738 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.167674 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:17.167738 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.167724 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:17.167919 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.167738 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:17.167919 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.167788 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-credential-socket" pod "b6fcb72e-f1e8-4e88-ac56-fde05b65847c" (UID: "b6fcb72e-f1e8-4e88-ac56-fde05b65847c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:17.168100 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.168077 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-socket" pod "b6fcb72e-f1e8-4e88-ac56-fde05b65847c" (UID: "b6fcb72e-f1e8-4e88-ac56-fde05b65847c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:17.169435 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.169380 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-token" pod "b6fcb72e-f1e8-4e88-ac56-fde05b65847c" (UID: "b6fcb72e-f1e8-4e88-ac56-fde05b65847c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:17.169545 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.169466 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-envoy" pod "b6fcb72e-f1e8-4e88-ac56-fde05b65847c" (UID: "b6fcb72e-f1e8-4e88-ac56-fde05b65847c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:17.169730 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.169697 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-podinfo" pod "b6fcb72e-f1e8-4e88-ac56-fde05b65847c" (UID: "b6fcb72e-f1e8-4e88-ac56-fde05b65847c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:59:17.169932 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.169908 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-kube-api-access-xftvk" pod "b6fcb72e-f1e8-4e88-ac56-fde05b65847c" (UID: "b6fcb72e-f1e8-4e88-ac56-fde05b65847c"). InnerVolumeSpecName "kube-api-access-xftvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:17.273122 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.273087 4541 generic.go:356] "Generic (PLEG): container finished" podID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerID="6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7" exitCode=0 Nov 03 13:59:17.273286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.273152 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" event={"ID":"b6fcb72e-f1e8-4e88-ac56-fde05b65847c","Type":"ContainerDied","Data":"6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7"} Nov 03 13:59:17.273286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.273220 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" event={"ID":"b6fcb72e-f1e8-4e88-ac56-fde05b65847c","Type":"ContainerDied","Data":"47fc4b51baf16d791a961198fcf92c48e5c0028813734dde8691d3b8aa953cd0"} Nov 03 13:59:17.273286 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.273242 4541 scope.go:122] "RemoveContainer" containerID="6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7" Nov 03 13:59:17.273391 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.273356 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m" Nov 03 13:59:17.276302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.275608 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:17.276302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.275640 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xftvk\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-kube-api-access-xftvk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:17.276302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.275656 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:17.276302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.275668 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:17.276302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.275682 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:17.276302 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.275694 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6fcb72e-f1e8-4e88-ac56-fde05b65847c-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:17.289521 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.289498 4541 scope.go:122] "RemoveContainer" containerID="6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7" Nov 03 13:59:17.289977 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:59:17.289929 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7\": container with ID starting with 6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7 not found: ID does not exist" containerID="6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7" Nov 03 13:59:17.290085 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.289989 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7"} err="failed to get container status \"6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7\": rpc error: code = NotFound desc = could not find container \"6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7\": container with ID starting with 6934852eaf07071fb8e6acef08fe1cb5e685d70f4f0c8f557f195230f83afbb7 not found: ID does not exist" Nov 03 13:59:17.432372 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.432333 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m"] Nov 03 13:59:17.437301 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:17.437241 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-2849m"] Nov 03 13:59:18.869498 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:18.869464 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" path="/var/lib/kubelet/pods/b6fcb72e-f1e8-4e88-ac56-fde05b65847c/volumes" Nov 03 13:59:25.447983 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:25.447948 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps"] Nov 03 13:59:29.274976 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:29.274946 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:59:29.280173 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:29.280147 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 13:59:40.468204 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.468164 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" podUID="2bb00b65-8ce6-43ec-939e-ecdded328aaf" containerName="main" containerID="cri-o://8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406" gracePeriod=45 Nov 03 13:59:40.475046 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.475012 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" podUID="2bb00b65-8ce6-43ec-939e-ecdded328aaf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.116:8000/ready\": dial tcp 10.133.0.116:8000: connect: connection refused" Nov 03 13:59:40.733896 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.733836 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:59:40.841385 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.841350 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-home\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-home\") pod \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " Nov 03 13:59:40.841557 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.841395 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-model-cache\") pod \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " Nov 03 13:59:40.841557 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.841460 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tls-certs\") pod \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " Nov 03 13:59:40.841557 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.841504 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-dshm\") pod \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " Nov 03 13:59:40.841557 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.841521 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tmp-dir\") pod \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " Nov 03 13:59:40.841739 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.841573 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb00b65-8ce6-43ec-939e-ecdded328aaf-kube-api-access-w88xx\" (UniqueName: \"kubernetes.io/projected/2bb00b65-8ce6-43ec-939e-ecdded328aaf-kube-api-access-w88xx\") pod \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\" (UID: \"2bb00b65-8ce6-43ec-939e-ecdded328aaf\") " Nov 03 13:59:40.841739 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.841675 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-home" pod "2bb00b65-8ce6-43ec-939e-ecdded328aaf" (UID: "2bb00b65-8ce6-43ec-939e-ecdded328aaf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:40.841739 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.841681 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-model-cache" pod "2bb00b65-8ce6-43ec-939e-ecdded328aaf" (UID: "2bb00b65-8ce6-43ec-939e-ecdded328aaf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:40.841948 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.841880 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tmp-dir" pod "2bb00b65-8ce6-43ec-939e-ecdded328aaf" (UID: "2bb00b65-8ce6-43ec-939e-ecdded328aaf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:40.842146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.842119 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:40.842146 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.842143 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:40.842317 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.842155 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:40.843858 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.843839 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb00b65-8ce6-43ec-939e-ecdded328aaf-kube-api-access-w88xx" pod "2bb00b65-8ce6-43ec-939e-ecdded328aaf" (UID: "2bb00b65-8ce6-43ec-939e-ecdded328aaf"). InnerVolumeSpecName "kube-api-access-w88xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:59:40.844174 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.844146 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-dshm" pod "2bb00b65-8ce6-43ec-939e-ecdded328aaf" (UID: "2bb00b65-8ce6-43ec-939e-ecdded328aaf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:59:40.844270 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.844152 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tls-certs" pod "2bb00b65-8ce6-43ec-939e-ecdded328aaf" (UID: "2bb00b65-8ce6-43ec-939e-ecdded328aaf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:59:40.943035 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.943003 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb00b65-8ce6-43ec-939e-ecdded328aaf-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:40.943035 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.943031 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb00b65-8ce6-43ec-939e-ecdded328aaf-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:40.943035 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:40.943042 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w88xx\" (UniqueName: \"kubernetes.io/projected/2bb00b65-8ce6-43ec-939e-ecdded328aaf-kube-api-access-w88xx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:59:41.413013 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:41.412979 4541 generic.go:356] "Generic (PLEG): container finished" podID="2bb00b65-8ce6-43ec-939e-ecdded328aaf" containerID="8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406" exitCode=0 Nov 03 13:59:41.413218 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:41.413034 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" Nov 03 13:59:41.413218 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:41.413068 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" event={"ID":"2bb00b65-8ce6-43ec-939e-ecdded328aaf","Type":"ContainerDied","Data":"8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406"} Nov 03 13:59:41.413218 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:41.413113 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps" event={"ID":"2bb00b65-8ce6-43ec-939e-ecdded328aaf","Type":"ContainerDied","Data":"373fd194cd7513ceeeff47eeef34d35e3db893cb0010fe87f76a369037fb7660"} Nov 03 13:59:41.413218 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:41.413130 4541 scope.go:122] "RemoveContainer" containerID="8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406" Nov 03 13:59:41.422937 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:41.422901 4541 scope.go:122] "RemoveContainer" containerID="8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406" Nov 03 13:59:41.423435 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:59:41.423406 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406\": container with ID starting with 8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406 not found: ID does not exist" containerID="8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406" Nov 03 13:59:41.423533 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:41.423439 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406"} err="failed to get container status \"8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406\": rpc error: code = NotFound desc = could not find container \"8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406\": container with ID starting with 8cbde3dfde2b1caba877519f0113ecc73b34da5956c7f589cc6261789a0a9406 not found: ID does not exist" Nov 03 13:59:41.576999 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:41.576964 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps"] Nov 03 13:59:41.584814 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:41.584747 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5faf5d/auth-test-model-5faf5d-kserve-7468d547d5-mzjps"] Nov 03 13:59:42.868200 ip-10-0-132-174 kubenswrapper[4541]: I1103 13:59:42.868169 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb00b65-8ce6-43ec-939e-ecdded328aaf" path="/var/lib/kubelet/pods/2bb00b65-8ce6-43ec-939e-ecdded328aaf/volumes" Nov 03 13:59:46.889290 ip-10-0-132-174 kubenswrapper[4541]: E1103 13:59:46.889251 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:00:00.202901 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.202864 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl"] Nov 03 14:00:00.203426 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.203411 4541 cpu_manager.go:414] "containerMap: removing container" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" Nov 03 14:00:00.203471 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.203426 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" Nov 03 14:00:00.203471 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.203449 4541 cpu_manager.go:414] "containerMap: removing container" podUID="2bb00b65-8ce6-43ec-939e-ecdded328aaf" containerName="main" Nov 03 14:00:00.203471 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.203454 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb00b65-8ce6-43ec-939e-ecdded328aaf" containerName="main" Nov 03 14:00:00.203562 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.203479 4541 cpu_manager.go:414] "containerMap: removing container" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" Nov 03 14:00:00.203562 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.203484 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" Nov 03 14:00:00.203639 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.203586 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6fcb72e-f1e8-4e88-ac56-fde05b65847c" containerName="istio-proxy" Nov 03 14:00:00.203639 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.203596 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2fafe5-bc7b-4291-943c-c501430b7cf4" containerName="istio-proxy" Nov 03 14:00:00.203639 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.203605 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb00b65-8ce6-43ec-939e-ecdded328aaf" containerName="main" Nov 03 14:00:00.208571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.208550 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:00.222828 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.222804 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v"] Nov 03 14:00:00.226467 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.226451 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:00.311953 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.311921 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl"] Nov 03 14:00:00.354514 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.354473 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:00:00.358287 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.358254 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v"] Nov 03 14:00:00.374046 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.374014 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nxfq\" (UniqueName: \"kubernetes.io/projected/690eb9ee-0477-4fde-b55d-9831c9896c79-kube-api-access-5nxfq\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl\" (UID: \"690eb9ee-0477-4fde-b55d-9831c9896c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:00.374236 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.374213 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b933103-a871-4a09-8a98-d4763bb23566-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v\" (UID: \"3b933103-a871-4a09-8a98-d4763bb23566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:00.374303 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.374255 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f48b\" (UniqueName: \"kubernetes.io/projected/3b933103-a871-4a09-8a98-d4763bb23566-kube-api-access-9f48b\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v\" (UID: \"3b933103-a871-4a09-8a98-d4763bb23566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:00.374461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.374431 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690eb9ee-0477-4fde-b55d-9831c9896c79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl\" (UID: \"690eb9ee-0477-4fde-b55d-9831c9896c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:00.475960 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.475863 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b933103-a871-4a09-8a98-d4763bb23566-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v\" (UID: \"3b933103-a871-4a09-8a98-d4763bb23566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:00.475960 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.475911 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f48b\" (UniqueName: \"kubernetes.io/projected/3b933103-a871-4a09-8a98-d4763bb23566-kube-api-access-9f48b\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v\" (UID: \"3b933103-a871-4a09-8a98-d4763bb23566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:00.476204 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.476018 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690eb9ee-0477-4fde-b55d-9831c9896c79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl\" (UID: \"690eb9ee-0477-4fde-b55d-9831c9896c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:00.476204 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.476056 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nxfq\" (UniqueName: \"kubernetes.io/projected/690eb9ee-0477-4fde-b55d-9831c9896c79-kube-api-access-5nxfq\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl\" (UID: \"690eb9ee-0477-4fde-b55d-9831c9896c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:00.476525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.476502 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b933103-a871-4a09-8a98-d4763bb23566-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v\" (UID: \"3b933103-a871-4a09-8a98-d4763bb23566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:00.476686 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.476666 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690eb9ee-0477-4fde-b55d-9831c9896c79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl\" (UID: \"690eb9ee-0477-4fde-b55d-9831c9896c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:00.534552 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.534521 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nxfq\" (UniqueName: \"kubernetes.io/projected/690eb9ee-0477-4fde-b55d-9831c9896c79-kube-api-access-5nxfq\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl\" (UID: \"690eb9ee-0477-4fde-b55d-9831c9896c79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:00.534721 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.534698 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f48b\" (UniqueName: \"kubernetes.io/projected/3b933103-a871-4a09-8a98-d4763bb23566-kube-api-access-9f48b\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v\" (UID: \"3b933103-a871-4a09-8a98-d4763bb23566\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:00.539402 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.539378 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:00.822055 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.821985 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:00.918953 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:00.918920 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v"] Nov 03 14:00:00.920678 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:00:00.920648 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b933103_a871_4a09_8a98_d4763bb23566.slice/crio-2e3952671ecf6676c6b423ff4204f7287db0dfc791238aece0c94b8e4a46e5c9 WatchSource:0}: Error finding container 2e3952671ecf6676c6b423ff4204f7287db0dfc791238aece0c94b8e4a46e5c9: Status 404 returned error can't find the container with id 2e3952671ecf6676c6b423ff4204f7287db0dfc791238aece0c94b8e4a46e5c9 Nov 03 14:00:01.189511 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:01.189483 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl"] Nov 03 14:00:01.191157 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:00:01.191130 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690eb9ee_0477_4fde_b55d_9831c9896c79.slice/crio-8a3239c49d2aafc2de2a798519498a2074cfca1b0325075b570b280c5a26b7be WatchSource:0}: Error finding container 8a3239c49d2aafc2de2a798519498a2074cfca1b0325075b570b280c5a26b7be: Status 404 returned error can't find the container with id 8a3239c49d2aafc2de2a798519498a2074cfca1b0325075b570b280c5a26b7be Nov 03 14:00:01.539169 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:01.539135 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" event={"ID":"3b933103-a871-4a09-8a98-d4763bb23566","Type":"ContainerStarted","Data":"a7609f449188c06368963b61c4655fd51ff4c6069cb399e5a3458091ee13fa03"} Nov 03 14:00:01.539620 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:01.539190 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" event={"ID":"3b933103-a871-4a09-8a98-d4763bb23566","Type":"ContainerStarted","Data":"2e3952671ecf6676c6b423ff4204f7287db0dfc791238aece0c94b8e4a46e5c9"} Nov 03 14:00:01.540880 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:01.540858 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" event={"ID":"690eb9ee-0477-4fde-b55d-9831c9896c79","Type":"ContainerStarted","Data":"3f01998b134dfbef72dae7dbb6c3e09fb3264dcf26cbcbc8b482241c06aebac6"} Nov 03 14:00:01.540978 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:01.540897 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" event={"ID":"690eb9ee-0477-4fde-b55d-9831c9896c79","Type":"ContainerStarted","Data":"8a3239c49d2aafc2de2a798519498a2074cfca1b0325075b570b280c5a26b7be"} Nov 03 14:00:01.653666 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:01.653611 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" podStartSLOduration=1.653596748 podStartE2EDuration="1.653596748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:01.651970369 +0000 UTC m=+3332.965942228" watchObservedRunningTime="2026-11-03 14:00:01.653596748 +0000 UTC m=+3332.967568609" Nov 03 14:00:01.767649 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:01.767591 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" podStartSLOduration=1.767571581 podStartE2EDuration="1.767571581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:01.76561351 +0000 UTC m=+3333.079585370" watchObservedRunningTime="2026-11-03 14:00:01.767571581 +0000 UTC m=+3333.081543442" Nov 03 14:00:02.086005 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:02.085968 4541 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690eb9ee_0477_4fde_b55d_9831c9896c79.slice/crio-conmon-3f01998b134dfbef72dae7dbb6c3e09fb3264dcf26cbcbc8b482241c06aebac6.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:00:02.548423 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:02.548384 4541 generic.go:356] "Generic (PLEG): container finished" podID="3b933103-a871-4a09-8a98-d4763bb23566" containerID="a7609f449188c06368963b61c4655fd51ff4c6069cb399e5a3458091ee13fa03" exitCode=60 Nov 03 14:00:02.548872 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:02.548468 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" event={"ID":"3b933103-a871-4a09-8a98-d4763bb23566","Type":"ContainerDied","Data":"a7609f449188c06368963b61c4655fd51ff4c6069cb399e5a3458091ee13fa03"} Nov 03 14:00:02.548872 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:02.548728 4541 scope.go:122] "RemoveContainer" containerID="a7609f449188c06368963b61c4655fd51ff4c6069cb399e5a3458091ee13fa03" Nov 03 14:00:02.550735 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:02.550713 4541 generic.go:356] "Generic (PLEG): container finished" podID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerID="3f01998b134dfbef72dae7dbb6c3e09fb3264dcf26cbcbc8b482241c06aebac6" exitCode=60 Nov 03 14:00:02.550869 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:02.550812 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" event={"ID":"690eb9ee-0477-4fde-b55d-9831c9896c79","Type":"ContainerDied","Data":"3f01998b134dfbef72dae7dbb6c3e09fb3264dcf26cbcbc8b482241c06aebac6"} Nov 03 14:00:02.550990 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:02.550975 4541 scope.go:122] "RemoveContainer" containerID="3f01998b134dfbef72dae7dbb6c3e09fb3264dcf26cbcbc8b482241c06aebac6" Nov 03 14:00:03.561901 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:03.561861 4541 generic.go:356] "Generic (PLEG): container finished" podID="3b933103-a871-4a09-8a98-d4763bb23566" containerID="529a30c9752174b91f33b7d4c881b3826af434e927f5d5c74cff69987f78271e" exitCode=60 Nov 03 14:00:03.562272 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:03.561923 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" event={"ID":"3b933103-a871-4a09-8a98-d4763bb23566","Type":"ContainerDied","Data":"529a30c9752174b91f33b7d4c881b3826af434e927f5d5c74cff69987f78271e"} Nov 03 14:00:03.562272 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:03.562017 4541 scope.go:122] "RemoveContainer" containerID="a7609f449188c06368963b61c4655fd51ff4c6069cb399e5a3458091ee13fa03" Nov 03 14:00:03.562272 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:03.562260 4541 scope.go:122] "RemoveContainer" containerID="529a30c9752174b91f33b7d4c881b3826af434e927f5d5c74cff69987f78271e" Nov 03 14:00:03.562602 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:03.562576 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v_odh-ai-gateway-infra(3b933103-a871-4a09-8a98-d4763bb23566)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" podUID="3b933103-a871-4a09-8a98-d4763bb23566" Nov 03 14:00:03.564924 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:03.564904 4541 generic.go:356] "Generic (PLEG): container finished" podID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerID="fe69170481eb49fd537247e67e1c52e0883ebc66a749b148440c2c2fa72cd4a6" exitCode=60 Nov 03 14:00:03.565013 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:03.564969 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" event={"ID":"690eb9ee-0477-4fde-b55d-9831c9896c79","Type":"ContainerDied","Data":"fe69170481eb49fd537247e67e1c52e0883ebc66a749b148440c2c2fa72cd4a6"} Nov 03 14:00:03.565199 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:03.565186 4541 scope.go:122] "RemoveContainer" containerID="fe69170481eb49fd537247e67e1c52e0883ebc66a749b148440c2c2fa72cd4a6" Nov 03 14:00:03.565418 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:03.565402 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl_odh-ai-gateway-infra(690eb9ee-0477-4fde-b55d-9831c9896c79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" Nov 03 14:00:03.573884 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:03.573867 4541 scope.go:122] "RemoveContainer" containerID="3f01998b134dfbef72dae7dbb6c3e09fb3264dcf26cbcbc8b482241c06aebac6" Nov 03 14:00:04.572955 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:04.572929 4541 scope.go:122] "RemoveContainer" containerID="fe69170481eb49fd537247e67e1c52e0883ebc66a749b148440c2c2fa72cd4a6" Nov 03 14:00:04.573424 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:04.573218 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl_odh-ai-gateway-infra(690eb9ee-0477-4fde-b55d-9831c9896c79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" Nov 03 14:00:04.574701 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:04.574678 4541 scope.go:122] "RemoveContainer" containerID="529a30c9752174b91f33b7d4c881b3826af434e927f5d5c74cff69987f78271e" Nov 03 14:00:04.574966 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:04.574944 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v_odh-ai-gateway-infra(3b933103-a871-4a09-8a98-d4763bb23566)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" podUID="3b933103-a871-4a09-8a98-d4763bb23566" Nov 03 14:00:10.826898 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.826865 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292"] Nov 03 14:00:10.832116 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.832090 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:10.885492 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.885463 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j6nw\" (UniqueName: \"kubernetes.io/projected/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-kube-api-access-2j6nw\") pod \"payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292\" (UID: \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:10.885644 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.885551 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292"] Nov 03 14:00:10.885758 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.885721 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292\" (UID: \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:10.928999 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.928970 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c4e398\"" Nov 03 14:00:10.929130 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.929071 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c4e398-dockercfg-77rnl\"" Nov 03 14:00:10.986717 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.986677 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292\" (UID: \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:10.986929 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.986811 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j6nw\" (UniqueName: \"kubernetes.io/projected/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-kube-api-access-2j6nw\") pod \"payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292\" (UID: \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:10.987457 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:10.987435 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292\" (UID: \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:11.041881 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:11.041849 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j6nw\" (UniqueName: \"kubernetes.io/projected/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-kube-api-access-2j6nw\") pod \"payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292\" (UID: \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:11.145376 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:11.145291 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:11.518050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:11.518020 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292"] Nov 03 14:00:11.518782 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:00:11.518709 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1e48d4_0e3a_4fef_b4da_dd28a0f594d3.slice/crio-e0453e41626e930b3b9bbfb74a513a7f03050c6e711e1f9d26a39537c6ab0a7e WatchSource:0}: Error finding container e0453e41626e930b3b9bbfb74a513a7f03050c6e711e1f9d26a39537c6ab0a7e: Status 404 returned error can't find the container with id e0453e41626e930b3b9bbfb74a513a7f03050c6e711e1f9d26a39537c6ab0a7e Nov 03 14:00:11.616334 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:11.616302 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" event={"ID":"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3","Type":"ContainerStarted","Data":"83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6"} Nov 03 14:00:11.616436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:11.616355 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" event={"ID":"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3","Type":"ContainerStarted","Data":"e0453e41626e930b3b9bbfb74a513a7f03050c6e711e1f9d26a39537c6ab0a7e"} Nov 03 14:00:11.616436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:11.616412 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:11.725861 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:11.725816 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" podStartSLOduration=1.7258025890000002 podStartE2EDuration="1.725802589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:11.72475887 +0000 UTC m=+3343.038730727" watchObservedRunningTime="2026-11-03 14:00:11.725802589 +0000 UTC m=+3343.039774448" Nov 03 14:00:14.861436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:14.861406 4541 scope.go:122] "RemoveContainer" containerID="fe69170481eb49fd537247e67e1c52e0883ebc66a749b148440c2c2fa72cd4a6" Nov 03 14:00:15.643232 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:15.643199 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" event={"ID":"690eb9ee-0477-4fde-b55d-9831c9896c79","Type":"ContainerStarted","Data":"7745746dee033906377eea3eeda54bc1bd53a6dee38d8de32ffa720c836d1ae6"} Nov 03 14:00:15.657513 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:15.657482 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm"] Nov 03 14:00:16.654566 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:16.654532 4541 generic.go:356] "Generic (PLEG): container finished" podID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerID="7745746dee033906377eea3eeda54bc1bd53a6dee38d8de32ffa720c836d1ae6" exitCode=60 Nov 03 14:00:16.655316 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:16.654614 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" event={"ID":"690eb9ee-0477-4fde-b55d-9831c9896c79","Type":"ContainerDied","Data":"7745746dee033906377eea3eeda54bc1bd53a6dee38d8de32ffa720c836d1ae6"} Nov 03 14:00:16.655316 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:16.654724 4541 scope.go:122] "RemoveContainer" containerID="fe69170481eb49fd537247e67e1c52e0883ebc66a749b148440c2c2fa72cd4a6" Nov 03 14:00:16.655316 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:16.655072 4541 scope.go:122] "RemoveContainer" containerID="7745746dee033906377eea3eeda54bc1bd53a6dee38d8de32ffa720c836d1ae6" Nov 03 14:00:16.655316 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:16.655311 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl_odh-ai-gateway-infra(690eb9ee-0477-4fde-b55d-9831c9896c79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" Nov 03 14:00:16.774004 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:16.773968 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl"] Nov 03 14:00:16.888655 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:16.888621 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:00:17.798422 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:17.798394 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:17.858870 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:17.858846 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690eb9ee-0477-4fde-b55d-9831c9896c79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690eb9ee-0477-4fde-b55d-9831c9896c79-maas-api-service-ca\") pod \"690eb9ee-0477-4fde-b55d-9831c9896c79\" (UID: \"690eb9ee-0477-4fde-b55d-9831c9896c79\") " Nov 03 14:00:17.859070 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:17.859054 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690eb9ee-0477-4fde-b55d-9831c9896c79-kube-api-access-5nxfq\" (UniqueName: \"kubernetes.io/projected/690eb9ee-0477-4fde-b55d-9831c9896c79-kube-api-access-5nxfq\") pod \"690eb9ee-0477-4fde-b55d-9831c9896c79\" (UID: \"690eb9ee-0477-4fde-b55d-9831c9896c79\") " Nov 03 14:00:17.859207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:17.859181 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690eb9ee-0477-4fde-b55d-9831c9896c79-maas-api-service-ca" pod "690eb9ee-0477-4fde-b55d-9831c9896c79" (UID: "690eb9ee-0477-4fde-b55d-9831c9896c79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:17.859417 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:17.859402 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/690eb9ee-0477-4fde-b55d-9831c9896c79-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:17.861101 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:17.861080 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690eb9ee-0477-4fde-b55d-9831c9896c79-kube-api-access-5nxfq" pod "690eb9ee-0477-4fde-b55d-9831c9896c79" (UID: "690eb9ee-0477-4fde-b55d-9831c9896c79"). InnerVolumeSpecName "kube-api-access-5nxfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:17.861400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:17.861386 4541 scope.go:122] "RemoveContainer" containerID="529a30c9752174b91f33b7d4c881b3826af434e927f5d5c74cff69987f78271e" Nov 03 14:00:17.960980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:17.960905 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nxfq\" (UniqueName: \"kubernetes.io/projected/690eb9ee-0477-4fde-b55d-9831c9896c79-kube-api-access-5nxfq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.668923 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:18.668893 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" Nov 03 14:00:18.669096 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:18.668917 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl" event={"ID":"690eb9ee-0477-4fde-b55d-9831c9896c79","Type":"ContainerDied","Data":"8a3239c49d2aafc2de2a798519498a2074cfca1b0325075b570b280c5a26b7be"} Nov 03 14:00:18.669096 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:18.668975 4541 scope.go:122] "RemoveContainer" containerID="7745746dee033906377eea3eeda54bc1bd53a6dee38d8de32ffa720c836d1ae6" Nov 03 14:00:18.671519 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:18.671495 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" event={"ID":"3b933103-a871-4a09-8a98-d4763bb23566","Type":"ContainerStarted","Data":"1fef407b8011506ee0c9554fbf21ca378d2b6e6867b57ae0639f062462d35404"} Nov 03 14:00:18.825201 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:18.825157 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl"] Nov 03 14:00:18.877144 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:18.877114 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895240-pf7xl"] Nov 03 14:00:19.679092 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:19.679060 4541 generic.go:356] "Generic (PLEG): container finished" podID="3b933103-a871-4a09-8a98-d4763bb23566" containerID="1fef407b8011506ee0c9554fbf21ca378d2b6e6867b57ae0639f062462d35404" exitCode=60 Nov 03 14:00:19.679286 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:19.679136 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" event={"ID":"3b933103-a871-4a09-8a98-d4763bb23566","Type":"ContainerDied","Data":"1fef407b8011506ee0c9554fbf21ca378d2b6e6867b57ae0639f062462d35404"} Nov 03 14:00:19.679286 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:19.679201 4541 scope.go:122] "RemoveContainer" containerID="529a30c9752174b91f33b7d4c881b3826af434e927f5d5c74cff69987f78271e" Nov 03 14:00:19.679552 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:19.679528 4541 scope.go:122] "RemoveContainer" containerID="1fef407b8011506ee0c9554fbf21ca378d2b6e6867b57ae0639f062462d35404" Nov 03 14:00:19.679902 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:19.679875 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v_odh-ai-gateway-infra(3b933103-a871-4a09-8a98-d4763bb23566)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" podUID="3b933103-a871-4a09-8a98-d4763bb23566" Nov 03 14:00:19.943499 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:19.943421 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v"] Nov 03 14:00:20.838722 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:20.838699 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:20.868784 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:20.868749 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" path="/var/lib/kubelet/pods/690eb9ee-0477-4fde-b55d-9831c9896c79/volumes" Nov 03 14:00:20.894553 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:20.894529 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b933103-a871-4a09-8a98-d4763bb23566-kube-api-access-9f48b\" (UniqueName: \"kubernetes.io/projected/3b933103-a871-4a09-8a98-d4763bb23566-kube-api-access-9f48b\") pod \"3b933103-a871-4a09-8a98-d4763bb23566\" (UID: \"3b933103-a871-4a09-8a98-d4763bb23566\") " Nov 03 14:00:20.894673 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:20.894583 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b933103-a871-4a09-8a98-d4763bb23566-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b933103-a871-4a09-8a98-d4763bb23566-maas-api-service-ca\") pod \"3b933103-a871-4a09-8a98-d4763bb23566\" (UID: \"3b933103-a871-4a09-8a98-d4763bb23566\") " Nov 03 14:00:20.894988 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:20.894957 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b933103-a871-4a09-8a98-d4763bb23566-maas-api-service-ca" pod "3b933103-a871-4a09-8a98-d4763bb23566" (UID: "3b933103-a871-4a09-8a98-d4763bb23566"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:20.896569 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:20.896548 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b933103-a871-4a09-8a98-d4763bb23566-kube-api-access-9f48b" pod "3b933103-a871-4a09-8a98-d4763bb23566" (UID: "3b933103-a871-4a09-8a98-d4763bb23566"). InnerVolumeSpecName "kube-api-access-9f48b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:20.996202 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:20.996127 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f48b\" (UniqueName: \"kubernetes.io/projected/3b933103-a871-4a09-8a98-d4763bb23566-kube-api-access-9f48b\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:20.996202 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:20.996154 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b933103-a871-4a09-8a98-d4763bb23566-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:21.694643 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:21.694612 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" Nov 03 14:00:21.694805 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:21.694646 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v" event={"ID":"3b933103-a871-4a09-8a98-d4763bb23566","Type":"ContainerDied","Data":"2e3952671ecf6676c6b423ff4204f7287db0dfc791238aece0c94b8e4a46e5c9"} Nov 03 14:00:21.694805 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:21.694711 4541 scope.go:122] "RemoveContainer" containerID="1fef407b8011506ee0c9554fbf21ca378d2b6e6867b57ae0639f062462d35404" Nov 03 14:00:21.850972 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:21.850931 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v"] Nov 03 14:00:21.854756 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:21.854732 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895240-qdw8v"] Nov 03 14:00:22.626478 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:22.626437 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:00:22.870057 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:22.870025 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b933103-a871-4a09-8a98-d4763bb23566" path="/var/lib/kubelet/pods/3b933103-a871-4a09-8a98-d4763bb23566/volumes" Nov 03 14:00:30.677668 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:30.677624 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" podUID="42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" containerName="main" containerID="cri-o://24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de" gracePeriod=45 Nov 03 14:00:30.954008 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:30.953978 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 14:00:31.110623 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.110593 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-model-cache\") pod \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " Nov 03 14:00:31.110784 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.110727 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tmp-dir\") pod \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " Nov 03 14:00:31.110825 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.110817 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-home\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-home\") pod \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " Nov 03 14:00:31.110868 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.110837 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-model-cache" pod "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" (UID: "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:31.110930 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.110870 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tls-certs\" (UniqueName: \"kubernetes.io/secret/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tls-certs\") pod \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " Nov 03 14:00:31.110930 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.110891 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-dshm\") pod \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " Nov 03 14:00:31.110930 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.110910 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-kube-api-access-h4x76\" (UniqueName: \"kubernetes.io/projected/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-kube-api-access-h4x76\") pod \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\" (UID: \"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03\") " Nov 03 14:00:31.111102 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.111000 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tmp-dir" pod "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" (UID: "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:31.111148 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.111109 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-home" pod "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" (UID: "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:31.111312 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.111289 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:31.111312 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.111313 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:31.111508 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.111322 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:31.113194 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.113170 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-kube-api-access-h4x76" pod "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" (UID: "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03"). InnerVolumeSpecName "kube-api-access-h4x76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:31.113544 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.113517 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tls-certs" pod "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" (UID: "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:00:31.113544 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.113525 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-dshm" pod "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" (UID: "42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:31.212859 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.212798 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:31.212859 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.212822 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:31.212859 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.212832 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4x76\" (UniqueName: \"kubernetes.io/projected/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03-kube-api-access-h4x76\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:31.758027 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.757989 4541 generic.go:356] "Generic (PLEG): container finished" podID="42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" containerID="24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de" exitCode=0 Nov 03 14:00:31.758546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.758033 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" Nov 03 14:00:31.758546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.758071 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" event={"ID":"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03","Type":"ContainerDied","Data":"24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de"} Nov 03 14:00:31.758546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.758114 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" event={"ID":"42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03","Type":"ContainerDied","Data":"7cee4e21bb90ee5f72eccb014bd021a4e13d8964e9134cc01116c53006385f5c"} Nov 03 14:00:31.758546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.758132 4541 scope.go:122] "RemoveContainer" containerID="24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de" Nov 03 14:00:31.767991 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.767973 4541 scope.go:122] "RemoveContainer" containerID="24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de" Nov 03 14:00:31.768254 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:31.768228 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de\": container with ID starting with 24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de not found: ID does not exist" containerID="24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de" Nov 03 14:00:31.768303 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.768262 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de"} err="failed to get container status \"24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de\": rpc error: code = NotFound desc = could not find container \"24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de\": container with ID starting with 24cf17cd9784fa88cdd18749b97b4b04d1168f236554bbd0cfa71867510820de not found: ID does not exist" Nov 03 14:00:31.905645 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.905597 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm" podUID="42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.117:8000/ready\": context deadline exceeded" Nov 03 14:00:31.914349 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.914316 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm"] Nov 03 14:00:31.961407 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:31.961371 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-d0231d/auth-test-model-d0231d-kserve-dcd957cc-gr4cm"] Nov 03 14:00:32.868666 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:00:32.868631 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" path="/var/lib/kubelet/pods/42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03/volumes" Nov 03 14:00:46.890652 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:00:46.890617 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:01:16.890041 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:01:16.890003 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:01:25.737542 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.737502 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp"] Nov 03 14:01:25.738269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738252 4541 cpu_manager.go:414] "containerMap: removing container" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerName="cleanup" Nov 03 14:01:25.738314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738270 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerName="cleanup" Nov 03 14:01:25.738314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738289 4541 cpu_manager.go:414] "containerMap: removing container" podUID="42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" containerName="main" Nov 03 14:01:25.738314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738294 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" containerName="main" Nov 03 14:01:25.738314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738300 4541 cpu_manager.go:414] "containerMap: removing container" podUID="3b933103-a871-4a09-8a98-d4763bb23566" containerName="cleanup" Nov 03 14:01:25.738314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738306 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b933103-a871-4a09-8a98-d4763bb23566" containerName="cleanup" Nov 03 14:01:25.738502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738344 4541 cpu_manager.go:414] "containerMap: removing container" podUID="3b933103-a871-4a09-8a98-d4763bb23566" containerName="cleanup" Nov 03 14:01:25.738502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738349 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b933103-a871-4a09-8a98-d4763bb23566" containerName="cleanup" Nov 03 14:01:25.738502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738361 4541 cpu_manager.go:414] "containerMap: removing container" podUID="3b933103-a871-4a09-8a98-d4763bb23566" containerName="cleanup" Nov 03 14:01:25.738502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738365 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b933103-a871-4a09-8a98-d4763bb23566" containerName="cleanup" Nov 03 14:01:25.738502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738404 4541 cpu_manager.go:414] "containerMap: removing container" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerName="cleanup" Nov 03 14:01:25.738502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738411 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerName="cleanup" Nov 03 14:01:25.738752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738527 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b933103-a871-4a09-8a98-d4763bb23566" containerName="cleanup" Nov 03 14:01:25.738752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738543 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b933103-a871-4a09-8a98-d4763bb23566" containerName="cleanup" Nov 03 14:01:25.738752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738556 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerName="cleanup" Nov 03 14:01:25.738752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738564 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerName="cleanup" Nov 03 14:01:25.738752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.738573 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="42a7cbb7-e710-4b4a-9fd5-d0ce099d7a03" containerName="main" Nov 03 14:01:25.742040 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.742022 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.795402 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.795359 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-58f896-openshift-default-dockercfg-jwzdz\"" Nov 03 14:01:25.802876 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.802840 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-podinfo\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.802876 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.802899 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-envoy\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.803181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.802966 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-credential-socket\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.803181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.802992 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvr9b\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-kube-api-access-vvr9b\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.803181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.803029 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-certs\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.803181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.803065 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-token\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.803181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.803110 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-socket\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.803457 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.803187 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-data\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.803457 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.803237 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istiod-ca-cert\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.842641 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.842598 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp"] Nov 03 14:01:25.904468 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904429 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-socket\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.904675 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904515 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-data\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.904675 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904558 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istiod-ca-cert\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.904675 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904642 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-podinfo\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.904909 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904800 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-envoy\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.904909 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904890 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-credential-socket\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.905025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904923 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvr9b\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-kube-api-access-vvr9b\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.905025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904936 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-socket\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.905025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904963 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-data\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.905025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.904966 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-certs\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.905025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.905017 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-token\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.905296 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.905209 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-credential-socket\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.905445 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.905423 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-certs\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.905507 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.905416 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istiod-ca-cert\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.907485 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.907462 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-podinfo\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.907624 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.907570 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-envoy\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.989370 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.989297 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-token\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:25.995489 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:25.995460 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvr9b\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-kube-api-access-vvr9b\") pod \"e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:26.055620 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:26.055583 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:26.480701 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:26.480663 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp"] Nov 03 14:01:26.517741 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:01:26.517705 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8eb416_4743_4d2d_9b4f_6356ba098cae.slice/crio-009a4564deb177550544e36c8c1f908571bfe281a923dafba33d0ebd56694b6a WatchSource:0}: Error finding container 009a4564deb177550544e36c8c1f908571bfe281a923dafba33d0ebd56694b6a: Status 404 returned error can't find the container with id 009a4564deb177550544e36c8c1f908571bfe281a923dafba33d0ebd56694b6a Nov 03 14:01:26.520473 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:26.520441 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:01:26.520553 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:26.520511 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:01:26.520553 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:26.520540 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:01:27.074251 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:27.074216 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" event={"ID":"9a8eb416-4743-4d2d-9b4f-6356ba098cae","Type":"ContainerStarted","Data":"9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a"} Nov 03 14:01:27.074648 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:27.074274 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" event={"ID":"9a8eb416-4743-4d2d-9b4f-6356ba098cae","Type":"ContainerStarted","Data":"009a4564deb177550544e36c8c1f908571bfe281a923dafba33d0ebd56694b6a"} Nov 03 14:01:27.191671 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:27.191582 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" podStartSLOduration=2.191563827 podStartE2EDuration="2.191563827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:01:27.18928677 +0000 UTC m=+3418.503258635" watchObservedRunningTime="2026-11-03 14:01:27.191563827 +0000 UTC m=+3418.505535689" Nov 03 14:01:28.056243 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:28.056201 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:28.062139 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:28.062112 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:28.080468 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:28.080439 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:28.081293 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:28.081274 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:01:35.049493 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.049458 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht"] Nov 03 14:01:35.050074 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.050046 4541 cpu_manager.go:414] "containerMap: removing container" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerName="cleanup" Nov 03 14:01:35.050074 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.050069 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerName="cleanup" Nov 03 14:01:35.050269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.050236 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="690eb9ee-0477-4fde-b55d-9831c9896c79" containerName="cleanup" Nov 03 14:01:35.050269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.050259 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b933103-a871-4a09-8a98-d4763bb23566" containerName="cleanup" Nov 03 14:01:35.053666 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.053649 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:01:35.103420 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.103386 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-58f896\"" Nov 03 14:01:35.150481 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.150443 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-58f896-dockercfg-h6hpz\"" Nov 03 14:01:35.153524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.153505 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht"] Nov 03 14:01:35.206210 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.206177 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c49079f-1630-4b75-b516-4c3a03a328c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht\" (UID: \"8c49079f-1630-4b75-b516-4c3a03a328c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:01:35.206378 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.206228 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh5lj\" (UniqueName: \"kubernetes.io/projected/8c49079f-1630-4b75-b516-4c3a03a328c2-kube-api-access-kh5lj\") pod \"payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht\" (UID: \"8c49079f-1630-4b75-b516-4c3a03a328c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:01:35.317764 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.317665 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c49079f-1630-4b75-b516-4c3a03a328c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht\" (UID: \"8c49079f-1630-4b75-b516-4c3a03a328c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:01:35.317764 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.317745 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh5lj\" (UniqueName: \"kubernetes.io/projected/8c49079f-1630-4b75-b516-4c3a03a328c2-kube-api-access-kh5lj\") pod \"payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht\" (UID: \"8c49079f-1630-4b75-b516-4c3a03a328c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:01:35.318491 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.318467 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c49079f-1630-4b75-b516-4c3a03a328c2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht\" (UID: \"8c49079f-1630-4b75-b516-4c3a03a328c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:01:35.372509 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.372472 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh5lj\" (UniqueName: \"kubernetes.io/projected/8c49079f-1630-4b75-b516-4c3a03a328c2-kube-api-access-kh5lj\") pod \"payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht\" (UID: \"8c49079f-1630-4b75-b516-4c3a03a328c2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:01:35.666321 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:35.666290 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:01:36.033277 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:36.033183 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht"] Nov 03 14:01:36.034625 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:01:36.034592 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c49079f_1630_4b75_b516_4c3a03a328c2.slice/crio-35df672c8f6ebe9cf2a06a7fd3563a40350f617a4bc532104cebf65dae489ff4 WatchSource:0}: Error finding container 35df672c8f6ebe9cf2a06a7fd3563a40350f617a4bc532104cebf65dae489ff4: Status 404 returned error can't find the container with id 35df672c8f6ebe9cf2a06a7fd3563a40350f617a4bc532104cebf65dae489ff4 Nov 03 14:01:36.127094 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:36.127064 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" event={"ID":"8c49079f-1630-4b75-b516-4c3a03a328c2","Type":"ContainerStarted","Data":"aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad"} Nov 03 14:01:36.127525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:36.127122 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" event={"ID":"8c49079f-1630-4b75-b516-4c3a03a328c2","Type":"ContainerStarted","Data":"35df672c8f6ebe9cf2a06a7fd3563a40350f617a4bc532104cebf65dae489ff4"} Nov 03 14:01:36.127525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:36.127290 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:01:36.235331 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:36.235280 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" podStartSLOduration=1.235266461 podStartE2EDuration="1.235266461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:01:36.234372578 +0000 UTC m=+3427.548344437" watchObservedRunningTime="2026-11-03 14:01:36.235266461 +0000 UTC m=+3427.549238320" Nov 03 14:01:46.890625 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:01:46.890587 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:01:47.136407 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:01:47.136375 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:02:16.890205 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:02:16.890121 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:02:17.984049 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.984006 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh"] Nov 03 14:02:17.988246 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.988218 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:17.995725 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.995696 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-data\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:17.995868 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.995740 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:17.995868 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.995851 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-credential-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:17.995960 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.995908 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-token\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:17.995960 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.995935 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bscj\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-kube-api-access-5bscj\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:17.996083 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.996066 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-envoy\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:17.996138 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.996125 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a34c955-2fbc-4f7b-8a19-7df762af539e-istiod-ca-cert\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:17.996204 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.996193 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-podinfo\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:17.996250 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:17.996216 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-certs\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.038413 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.038380 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c4e398-openshift-default-dockercfg-d4vg5\"" Nov 03 14:02:18.089667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.089631 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh"] Nov 03 14:02:18.097576 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.097545 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-envoy\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.097750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.097602 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a34c955-2fbc-4f7b-8a19-7df762af539e-istiod-ca-cert\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.097750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.097622 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-podinfo\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.097750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.097644 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-certs\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.097750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.097669 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-data\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.097750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.097687 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.097750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.097705 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-credential-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.097750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.097730 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-token\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.097750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.097746 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bscj\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-kube-api-access-5bscj\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.098704 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.098676 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.099596 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.098690 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-credential-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.099736 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.099137 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-data\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.099878 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.099152 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-certs\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.104830 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.099536 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a34c955-2fbc-4f7b-8a19-7df762af539e-istiod-ca-cert\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.104940 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.101153 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-envoy\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.104940 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.102227 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-podinfo\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.153245 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.153214 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-token\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.153411 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.153359 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bscj\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-kube-api-access-5bscj\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.302143 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.302061 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:18.721198 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.721169 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh"] Nov 03 14:02:18.722205 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:02:18.722173 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a34c955_2fbc_4f7b_8a19_7df762af539e.slice/crio-a8552cb7fdcb61fcd68bddf8da0554f0a62fcbd38e28dc5ead4ea5c5d3792db3 WatchSource:0}: Error finding container a8552cb7fdcb61fcd68bddf8da0554f0a62fcbd38e28dc5ead4ea5c5d3792db3: Status 404 returned error can't find the container with id a8552cb7fdcb61fcd68bddf8da0554f0a62fcbd38e28dc5ead4ea5c5d3792db3 Nov 03 14:02:18.724661 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.724625 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:02:18.724728 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.724705 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:02:18.724800 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:18.724735 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:02:19.389536 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:19.389496 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" event={"ID":"8a34c955-2fbc-4f7b-8a19-7df762af539e","Type":"ContainerStarted","Data":"d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627"} Nov 03 14:02:19.389993 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:19.389559 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" event={"ID":"8a34c955-2fbc-4f7b-8a19-7df762af539e","Type":"ContainerStarted","Data":"a8552cb7fdcb61fcd68bddf8da0554f0a62fcbd38e28dc5ead4ea5c5d3792db3"} Nov 03 14:02:19.508639 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:19.508546 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" podStartSLOduration=2.508525397 podStartE2EDuration="2.508525397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:19.504281566 +0000 UTC m=+3470.818253426" watchObservedRunningTime="2026-11-03 14:02:19.508525397 +0000 UTC m=+3470.822497258" Nov 03 14:02:20.303172 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:20.303139 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:20.304969 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:20.304942 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:02:20.305096 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:20.305002 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 14:02:21.302961 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:21.302917 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:02:21.303363 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:21.302966 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 14:02:22.302714 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:22.302677 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:02:22.302964 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:22.302734 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 14:02:23.303425 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:23.303377 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:02:23.303946 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:23.303441 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 14:02:24.302495 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:24.302454 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Nov 03 14:02:24.302715 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:24.302508 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Nov 03 14:02:25.307615 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:25.307583 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:25.308125 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:25.308038 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:25.308707 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:25.308688 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:02:39.994314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:39.994274 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j"] Nov 03 14:02:39.999966 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:39.999936 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:02:40.054132 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.054094 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j"] Nov 03 14:02:40.096182 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.096145 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d5d415-dockercfg-kwpq5\"" Nov 03 14:02:40.096182 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.096145 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d5d415\"" Nov 03 14:02:40.176406 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.176359 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrzq\" (UniqueName: \"kubernetes.io/projected/b3691312-271b-4ce7-aad6-fc8731f3f62b-kube-api-access-dlrzq\") pod \"payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j\" (UID: \"b3691312-271b-4ce7-aad6-fc8731f3f62b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:02:40.176610 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.176488 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3691312-271b-4ce7-aad6-fc8731f3f62b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j\" (UID: \"b3691312-271b-4ce7-aad6-fc8731f3f62b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:02:40.278123 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.278027 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3691312-271b-4ce7-aad6-fc8731f3f62b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j\" (UID: \"b3691312-271b-4ce7-aad6-fc8731f3f62b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:02:40.278297 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.278200 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrzq\" (UniqueName: \"kubernetes.io/projected/b3691312-271b-4ce7-aad6-fc8731f3f62b-kube-api-access-dlrzq\") pod \"payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j\" (UID: \"b3691312-271b-4ce7-aad6-fc8731f3f62b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:02:40.278877 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.278848 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3691312-271b-4ce7-aad6-fc8731f3f62b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j\" (UID: \"b3691312-271b-4ce7-aad6-fc8731f3f62b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:02:40.341502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.341462 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrzq\" (UniqueName: \"kubernetes.io/projected/b3691312-271b-4ce7-aad6-fc8731f3f62b-kube-api-access-dlrzq\") pod \"payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j\" (UID: \"b3691312-271b-4ce7-aad6-fc8731f3f62b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:02:40.619588 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:40.619501 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:02:41.001571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:41.001529 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j"] Nov 03 14:02:41.534007 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:41.533974 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" event={"ID":"b3691312-271b-4ce7-aad6-fc8731f3f62b","Type":"ContainerStarted","Data":"30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5"} Nov 03 14:02:41.534174 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:41.534024 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" event={"ID":"b3691312-271b-4ce7-aad6-fc8731f3f62b","Type":"ContainerStarted","Data":"069da0fba35de06c5e1c80d5a640e132a434ef19a3c2a0045dd16070f29b08fb"} Nov 03 14:02:41.534174 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:41.534055 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:02:41.644187 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:41.644134 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" podStartSLOduration=2.644121429 podStartE2EDuration="2.644121429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:41.641264828 +0000 UTC m=+3492.955236688" watchObservedRunningTime="2026-11-03 14:02:41.644121429 +0000 UTC m=+3492.958093289" Nov 03 14:02:46.890837 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:02:46.890792 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:02:52.543010 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:02:52.542979 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:03:03.502635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.502598 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg"] Nov 03 14:03:03.506900 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.506882 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.557217 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.557183 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-966a03-openshift-default-dockercfg-4wkwt\"" Nov 03 14:03:03.607941 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.607910 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg"] Nov 03 14:03:03.648830 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.648793 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-envoy\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.649000 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.648900 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-podinfo\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.649000 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.648962 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-token\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.649087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.649018 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794d684-2185-47cd-9c22-656bf45d1a4c-istiod-ca-cert\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.649087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.649066 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb5gr\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-kube-api-access-sb5gr\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.649181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.649121 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-certs\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.649228 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.649181 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-socket\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.649228 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.649224 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-credential-socket\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.649317 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.649261 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-data\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751011 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.750976 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-envoy\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751218 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751034 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-podinfo\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751218 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751077 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-token\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751218 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751098 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794d684-2185-47cd-9c22-656bf45d1a4c-istiod-ca-cert\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751399 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751365 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb5gr\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-kube-api-access-sb5gr\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751455 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751413 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-certs\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751513 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751460 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-socket\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751513 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751482 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-credential-socket\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751638 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751611 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-data\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751823 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751749 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-socket\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751919 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751841 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-credential-socket\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.751919 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751839 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-certs\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.752038 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.751968 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794d684-2185-47cd-9c22-656bf45d1a4c-istiod-ca-cert\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.752098 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.752067 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-data\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.753858 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.753804 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-envoy\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.754042 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.754024 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-podinfo\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.807334 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.807299 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-token\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.807506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.807415 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb5gr\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-kube-api-access-sb5gr\") pod \"e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:03.820857 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:03.820830 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:04.237203 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:04.237174 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg"] Nov 03 14:03:04.238813 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:03:04.238789 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6794d684_2185_47cd_9c22_656bf45d1a4c.slice/crio-87ba34cfb7eb39731cfb8c37a51dda0aaac58c43b3dc8f82da9da0aeb042b892 WatchSource:0}: Error finding container 87ba34cfb7eb39731cfb8c37a51dda0aaac58c43b3dc8f82da9da0aeb042b892: Status 404 returned error can't find the container with id 87ba34cfb7eb39731cfb8c37a51dda0aaac58c43b3dc8f82da9da0aeb042b892 Nov 03 14:03:04.241511 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:04.241475 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:03:04.241598 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:04.241544 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:03:04.241598 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:04.241573 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:03:04.677279 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:04.677235 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" event={"ID":"6794d684-2185-47cd-9c22-656bf45d1a4c","Type":"ContainerStarted","Data":"8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7"} Nov 03 14:03:04.677728 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:04.677294 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" event={"ID":"6794d684-2185-47cd-9c22-656bf45d1a4c","Type":"ContainerStarted","Data":"87ba34cfb7eb39731cfb8c37a51dda0aaac58c43b3dc8f82da9da0aeb042b892"} Nov 03 14:03:04.803057 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:04.803006 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" podStartSLOduration=1.802993137 podStartE2EDuration="1.802993137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:03:04.799128508 +0000 UTC m=+3516.113100368" watchObservedRunningTime="2026-11-03 14:03:04.802993137 +0000 UTC m=+3516.116964997" Nov 03 14:03:04.821734 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:04.821701 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:05.827165 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:05.827132 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:06.692974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:06.692944 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:06.698339 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:06.698316 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:03:12.822255 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:12.822207 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr"] Nov 03 14:03:12.826572 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:12.826553 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:12.876968 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:12.876928 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-966a03\"" Nov 03 14:03:12.880089 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:12.880058 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr"] Nov 03 14:03:12.923249 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:12.923213 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-966a03-dockercfg-dg4js\"" Nov 03 14:03:12.955343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:12.955306 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr\" (UID: \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:12.955521 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:12.955354 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhw5g\" (UniqueName: \"kubernetes.io/projected/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-kube-api-access-nhw5g\") pod \"payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr\" (UID: \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:13.056453 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.056413 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr\" (UID: \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:13.056453 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.056455 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhw5g\" (UniqueName: \"kubernetes.io/projected/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-kube-api-access-nhw5g\") pod \"payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr\" (UID: \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:13.057139 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.057119 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr\" (UID: \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:13.111127 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.111056 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhw5g\" (UniqueName: \"kubernetes.io/projected/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-kube-api-access-nhw5g\") pod \"payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr\" (UID: \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:13.141190 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.141162 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:13.511060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.511022 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr"] Nov 03 14:03:13.511422 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:03:13.511397 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode825ac61_1ada_4fc7_afb3_ac562d59e4d4.slice/crio-5f8da53879502bafca192c86fdc4266c7aad0525a9c9b6148ebad6b6a4695ccd WatchSource:0}: Error finding container 5f8da53879502bafca192c86fdc4266c7aad0525a9c9b6148ebad6b6a4695ccd: Status 404 returned error can't find the container with id 5f8da53879502bafca192c86fdc4266c7aad0525a9c9b6148ebad6b6a4695ccd Nov 03 14:03:13.736978 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.736939 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" event={"ID":"e825ac61-1ada-4fc7-afb3-ac562d59e4d4","Type":"ContainerStarted","Data":"9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee"} Nov 03 14:03:13.737146 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.737002 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" event={"ID":"e825ac61-1ada-4fc7-afb3-ac562d59e4d4","Type":"ContainerStarted","Data":"5f8da53879502bafca192c86fdc4266c7aad0525a9c9b6148ebad6b6a4695ccd"} Nov 03 14:03:13.737211 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.737169 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:13.891288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:13.891189 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" podStartSLOduration=1.891175794 podStartE2EDuration="1.891175794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:03:13.890908595 +0000 UTC m=+3525.204880456" watchObservedRunningTime="2026-11-03 14:03:13.891175794 +0000 UTC m=+3525.205147654" Nov 03 14:03:16.890804 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:03:16.890749 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:03:24.761144 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:24.761112 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:03:28.990695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:28.990217 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": read tcp 10.133.0.2:60154->10.133.0.122:15021: read: connection reset by peer" start-of-body= Nov 03 14:03:28.990695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:28.990274 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": read tcp 10.133.0.2:60154->10.133.0.122:15021: read: connection reset by peer" Nov 03 14:03:29.833995 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:29.833963 4541 generic.go:356] "Generic (PLEG): container finished" podID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerID="9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a" exitCode=0 Nov 03 14:03:29.834183 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:29.834040 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" event={"ID":"9a8eb416-4743-4d2d-9b4f-6356ba098cae","Type":"ContainerDied","Data":"9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a"} Nov 03 14:03:29.834430 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:29.834416 4541 scope.go:122] "RemoveContainer" containerID="9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a" Nov 03 14:03:29.835925 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:29.835887 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:03:29.836006 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:29.835975 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:03:29.836045 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:29.836009 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:03:30.868629 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:30.868592 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" event={"ID":"9a8eb416-4743-4d2d-9b4f-6356ba098cae","Type":"ContainerStarted","Data":"ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d"} Nov 03 14:03:31.056794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:31.056123 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:03:31.056794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:31.056251 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 14:03:31.056794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:31.056296 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 14:03:32.058795 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:32.058151 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 14:03:32.058795 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:32.058210 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 14:03:33.058794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:33.058093 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 14:03:33.058794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:33.058155 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 14:03:34.056794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:34.056119 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 14:03:34.056794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:34.056183 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 14:03:35.057807 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:35.056119 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 03 14:03:35.057807 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:35.056180 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 03 14:03:36.061004 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:36.060972 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:03:36.061492 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:36.061370 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:03:36.061967 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:03:36.061949 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:03:46.891530 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:03:46.891492 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:04:16.891730 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:04:16.891691 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:04:29.324624 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:29.324591 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:04:29.330857 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:29.330833 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:04:35.935415 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:35.935377 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht"] Nov 03 14:04:40.935917 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:40.935845 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" podUID="8c49079f-1630-4b75-b516-4c3a03a328c2" containerName="payload-processing" containerID="cri-o://aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad" gracePeriod=25 Nov 03 14:04:41.250024 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.249987 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:04:41.301113 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.301075 4541 generic.go:356] "Generic (PLEG): container finished" podID="8c49079f-1630-4b75-b516-4c3a03a328c2" containerID="aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad" exitCode=0 Nov 03 14:04:41.301113 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.301127 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" Nov 03 14:04:41.301375 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.301155 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" event={"ID":"8c49079f-1630-4b75-b516-4c3a03a328c2","Type":"ContainerDied","Data":"aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad"} Nov 03 14:04:41.301375 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.301212 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht" event={"ID":"8c49079f-1630-4b75-b516-4c3a03a328c2","Type":"ContainerDied","Data":"35df672c8f6ebe9cf2a06a7fd3563a40350f617a4bc532104cebf65dae489ff4"} Nov 03 14:04:41.301375 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.301236 4541 scope.go:122] "RemoveContainer" containerID="aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad" Nov 03 14:04:41.313856 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.313836 4541 scope.go:122] "RemoveContainer" containerID="aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad" Nov 03 14:04:41.314173 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:04:41.314147 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad\": container with ID starting with aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad not found: ID does not exist" containerID="aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad" Nov 03 14:04:41.314231 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.314185 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad"} err="failed to get container status \"aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad\": rpc error: code = NotFound desc = could not find container \"aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad\": container with ID starting with aa0a24939b820082c948e4e03bd4487172eba816ca98294f211aae0b172c97ad not found: ID does not exist" Nov 03 14:04:41.405349 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.405310 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c49079f-1630-4b75-b516-4c3a03a328c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c49079f-1630-4b75-b516-4c3a03a328c2-plugins-config-volume\") pod \"8c49079f-1630-4b75-b516-4c3a03a328c2\" (UID: \"8c49079f-1630-4b75-b516-4c3a03a328c2\") " Nov 03 14:04:41.405526 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.405485 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c49079f-1630-4b75-b516-4c3a03a328c2-kube-api-access-kh5lj\" (UniqueName: \"kubernetes.io/projected/8c49079f-1630-4b75-b516-4c3a03a328c2-kube-api-access-kh5lj\") pod \"8c49079f-1630-4b75-b516-4c3a03a328c2\" (UID: \"8c49079f-1630-4b75-b516-4c3a03a328c2\") " Nov 03 14:04:41.405783 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.405743 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c49079f-1630-4b75-b516-4c3a03a328c2-plugins-config-volume" pod "8c49079f-1630-4b75-b516-4c3a03a328c2" (UID: "8c49079f-1630-4b75-b516-4c3a03a328c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:04:41.405946 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.405930 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c49079f-1630-4b75-b516-4c3a03a328c2-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:41.407620 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.407595 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c49079f-1630-4b75-b516-4c3a03a328c2-kube-api-access-kh5lj" pod "8c49079f-1630-4b75-b516-4c3a03a328c2" (UID: "8c49079f-1630-4b75-b516-4c3a03a328c2"). InnerVolumeSpecName "kube-api-access-kh5lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:04:41.507748 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.507665 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh5lj\" (UniqueName: \"kubernetes.io/projected/8c49079f-1630-4b75-b516-4c3a03a328c2-kube-api-access-kh5lj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:41.718099 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.718061 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht"] Nov 03 14:04:41.723882 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:41.723851 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-58f896-74d85b8495-vktht"] Nov 03 14:04:42.869296 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:42.869260 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c49079f-1630-4b75-b516-4c3a03a328c2" path="/var/lib/kubelet/pods/8c49079f-1630-4b75-b516-4c3a03a328c2/volumes" Nov 03 14:04:46.890873 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:04:46.890829 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:04:51.793974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.793942 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798"] Nov 03 14:04:51.794713 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.794691 4541 cpu_manager.go:414] "containerMap: removing container" podUID="8c49079f-1630-4b75-b516-4c3a03a328c2" containerName="payload-processing" Nov 03 14:04:51.794813 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.794714 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c49079f-1630-4b75-b516-4c3a03a328c2" containerName="payload-processing" Nov 03 14:04:51.794866 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.794858 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c49079f-1630-4b75-b516-4c3a03a328c2" containerName="payload-processing" Nov 03 14:04:51.799638 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.799620 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:04:51.831065 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.831035 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e748d5bc-eabd-40a8-896e-4da00223a62a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798\" (UID: \"e748d5bc-eabd-40a8-896e-4da00223a62a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:04:51.831170 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.831073 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67v5d\" (UniqueName: \"kubernetes.io/projected/e748d5bc-eabd-40a8-896e-4da00223a62a-kube-api-access-67v5d\") pod \"payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798\" (UID: \"e748d5bc-eabd-40a8-896e-4da00223a62a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:04:51.851562 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.851529 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798"] Nov 03 14:04:51.895183 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.895148 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-aff1d2\"" Nov 03 14:04:51.895393 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.895303 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-aff1d2-dockercfg-mqn4s\"" Nov 03 14:04:51.932900 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.932862 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e748d5bc-eabd-40a8-896e-4da00223a62a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798\" (UID: \"e748d5bc-eabd-40a8-896e-4da00223a62a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:04:51.933078 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.932915 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67v5d\" (UniqueName: \"kubernetes.io/projected/e748d5bc-eabd-40a8-896e-4da00223a62a-kube-api-access-67v5d\") pod \"payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798\" (UID: \"e748d5bc-eabd-40a8-896e-4da00223a62a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:04:51.933763 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.933741 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e748d5bc-eabd-40a8-896e-4da00223a62a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798\" (UID: \"e748d5bc-eabd-40a8-896e-4da00223a62a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:04:51.943633 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.943609 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28"] Nov 03 14:04:51.947953 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.947934 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:04:51.989012 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:51.988982 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67v5d\" (UniqueName: \"kubernetes.io/projected/e748d5bc-eabd-40a8-896e-4da00223a62a-kube-api-access-67v5d\") pod \"payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798\" (UID: \"e748d5bc-eabd-40a8-896e-4da00223a62a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:04:52.001347 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.001321 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28"] Nov 03 14:04:52.033889 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.033862 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmjcz\" (UniqueName: \"kubernetes.io/projected/5348e4b3-1960-40c9-9f7b-ff07248f408c-kube-api-access-lmjcz\") pod \"payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28\" (UID: \"5348e4b3-1960-40c9-9f7b-ff07248f408c\") " pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:04:52.034055 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.033982 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5348e4b3-1960-40c9-9f7b-ff07248f408c-plugins-config-volume\") pod \"payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28\" (UID: \"5348e4b3-1960-40c9-9f7b-ff07248f408c\") " pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:04:52.115911 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.115835 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:04:52.135603 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.135578 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjcz\" (UniqueName: \"kubernetes.io/projected/5348e4b3-1960-40c9-9f7b-ff07248f408c-kube-api-access-lmjcz\") pod \"payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28\" (UID: \"5348e4b3-1960-40c9-9f7b-ff07248f408c\") " pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:04:52.135719 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.135693 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5348e4b3-1960-40c9-9f7b-ff07248f408c-plugins-config-volume\") pod \"payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28\" (UID: \"5348e4b3-1960-40c9-9f7b-ff07248f408c\") " pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:04:52.136584 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.136558 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5348e4b3-1960-40c9-9f7b-ff07248f408c-plugins-config-volume\") pod \"payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28\" (UID: \"5348e4b3-1960-40c9-9f7b-ff07248f408c\") " pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:04:52.191814 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.191322 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjcz\" (UniqueName: \"kubernetes.io/projected/5348e4b3-1960-40c9-9f7b-ff07248f408c-kube-api-access-lmjcz\") pod \"payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28\" (UID: \"5348e4b3-1960-40c9-9f7b-ff07248f408c\") " pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:04:52.262459 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.262425 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:04:52.485079 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.485051 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798"] Nov 03 14:04:52.486698 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:04:52.486670 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode748d5bc_eabd_40a8_896e_4da00223a62a.slice/crio-4a5c58b2d58102daca18a73e83eef451681b7ef81ee45574a9534ed99f9b28fb WatchSource:0}: Error finding container 4a5c58b2d58102daca18a73e83eef451681b7ef81ee45574a9534ed99f9b28fb: Status 404 returned error can't find the container with id 4a5c58b2d58102daca18a73e83eef451681b7ef81ee45574a9534ed99f9b28fb Nov 03 14:04:52.632574 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:52.632545 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28"] Nov 03 14:04:52.633065 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:04:52.633031 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5348e4b3_1960_40c9_9f7b_ff07248f408c.slice/crio-b55da2ef71ddd5b80333a866ba31324bb47f58ae7a3081c95a5b32dfc397c38e WatchSource:0}: Error finding container b55da2ef71ddd5b80333a866ba31324bb47f58ae7a3081c95a5b32dfc397c38e: Status 404 returned error can't find the container with id b55da2ef71ddd5b80333a866ba31324bb47f58ae7a3081c95a5b32dfc397c38e Nov 03 14:04:53.381224 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:53.381190 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" event={"ID":"5348e4b3-1960-40c9-9f7b-ff07248f408c","Type":"ContainerStarted","Data":"8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23"} Nov 03 14:04:53.381651 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:53.381253 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" event={"ID":"5348e4b3-1960-40c9-9f7b-ff07248f408c","Type":"ContainerStarted","Data":"b55da2ef71ddd5b80333a866ba31324bb47f58ae7a3081c95a5b32dfc397c38e"} Nov 03 14:04:53.381651 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:53.381284 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:04:53.382872 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:53.382846 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" event={"ID":"e748d5bc-eabd-40a8-896e-4da00223a62a","Type":"ContainerStarted","Data":"805af4ae64ba369b2144881fc9c1cf2b95fe6b096095acc564efb5955d9275b0"} Nov 03 14:04:53.383005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:53.382892 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" event={"ID":"e748d5bc-eabd-40a8-896e-4da00223a62a","Type":"ContainerStarted","Data":"4a5c58b2d58102daca18a73e83eef451681b7ef81ee45574a9534ed99f9b28fb"} Nov 03 14:04:53.383005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:53.382919 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:04:53.496871 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:53.496812 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" podStartSLOduration=2.496794187 podStartE2EDuration="2.496794187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:04:53.489687407 +0000 UTC m=+3624.803659269" watchObservedRunningTime="2026-11-03 14:04:53.496794187 +0000 UTC m=+3624.810766047" Nov 03 14:04:53.598855 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:04:53.598807 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" podStartSLOduration=2.598794681 podStartE2EDuration="2.598794681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:04:53.598734976 +0000 UTC m=+3624.912706836" watchObservedRunningTime="2026-11-03 14:04:53.598794681 +0000 UTC m=+3624.912766534" Nov 03 14:05:04.391527 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:04.391495 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:05:04.392039 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:04.391577 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:05:16.890334 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:05:16.890298 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:05:17.531326 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:17.531292 4541 generic.go:356] "Generic (PLEG): container finished" podID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerID="8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7" exitCode=0 Nov 03 14:05:17.531527 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:17.531366 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" event={"ID":"6794d684-2185-47cd-9c22-656bf45d1a4c","Type":"ContainerDied","Data":"8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7"} Nov 03 14:05:17.531865 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:17.531846 4541 scope.go:122] "RemoveContainer" containerID="8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7" Nov 03 14:05:17.533546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:17.533513 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:05:17.533656 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:17.533602 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:05:17.533726 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:17.533678 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:05:18.562312 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:18.562277 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" event={"ID":"6794d684-2185-47cd-9c22-656bf45d1a4c","Type":"ContainerStarted","Data":"69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406"} Nov 03 14:05:18.821502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:18.821422 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:05:18.822473 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:18.822451 4541 patch_prober.go:29] interesting pod/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 14:05:18.822533 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:18.822499 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 14:05:19.821374 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:19.821337 4541 patch_prober.go:29] interesting pod/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 14:05:19.821794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:19.821420 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 14:05:20.822020 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:20.821981 4541 patch_prober.go:29] interesting pod/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 14:05:20.822494 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:20.822050 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 14:05:21.692688 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:21.692654 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:05:21.821970 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:21.821888 4541 patch_prober.go:29] interesting pod/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 14:05:21.821970 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:21.821949 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 14:05:22.821667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:22.821633 4541 patch_prober.go:29] interesting pod/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 03 14:05:22.821869 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:22.821694 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 03 14:05:23.826752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:23.826720 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:05:23.827947 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:05:23.827925 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:05:46.892038 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:05:46.891992 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:06:13.395225 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:13.395187 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292"] Nov 03 14:06:16.891085 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:06:16.891048 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:06:18.396452 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.396390 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" podUID="9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3" containerName="payload-pre-processing" containerID="cri-o://83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6" gracePeriod=25 Nov 03 14:06:18.669318 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.669292 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:06:18.786244 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.786205 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-plugins-config-volume\") pod \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\" (UID: \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\") " Nov 03 14:06:18.786747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.786301 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-kube-api-access-2j6nw\" (UniqueName: \"kubernetes.io/projected/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-kube-api-access-2j6nw\") pod \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\" (UID: \"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3\") " Nov 03 14:06:18.787510 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.787482 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-plugins-config-volume" pod "9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3" (UID: "9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:18.805989 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.805936 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-kube-api-access-2j6nw" pod "9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3" (UID: "9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3"). InnerVolumeSpecName "kube-api-access-2j6nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:18.887628 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.887600 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:18.887628 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.887629 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j6nw\" (UniqueName: \"kubernetes.io/projected/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3-kube-api-access-2j6nw\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:18.942625 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.942548 4541 generic.go:356] "Generic (PLEG): container finished" podID="9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3" containerID="83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6" exitCode=0 Nov 03 14:06:18.942625 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.942595 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" Nov 03 14:06:18.942824 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.942630 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" event={"ID":"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3","Type":"ContainerDied","Data":"83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6"} Nov 03 14:06:18.942824 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.942683 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292" event={"ID":"9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3","Type":"ContainerDied","Data":"e0453e41626e930b3b9bbfb74a513a7f03050c6e711e1f9d26a39537c6ab0a7e"} Nov 03 14:06:18.942824 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.942712 4541 scope.go:122] "RemoveContainer" containerID="83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6" Nov 03 14:06:18.952471 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.952448 4541 scope.go:122] "RemoveContainer" containerID="83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6" Nov 03 14:06:18.952750 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:06:18.952721 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6\": container with ID starting with 83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6 not found: ID does not exist" containerID="83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6" Nov 03 14:06:18.952818 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:18.952762 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6"} err="failed to get container status \"83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6\": rpc error: code = NotFound desc = could not find container \"83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6\": container with ID starting with 83cbab44ca90cdb74f2faa327460747af10db14340ad81c494726509a40edab6 not found: ID does not exist" Nov 03 14:06:19.099589 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:19.099559 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292"] Nov 03 14:06:19.104891 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:19.104865 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c4e398-6dc4df7c9-67292"] Nov 03 14:06:19.623251 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:19.623193 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp"] Nov 03 14:06:19.623810 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:19.623548 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" containerID="cri-o://ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d" gracePeriod=30 Nov 03 14:06:20.870542 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:20.870503 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3" path="/var/lib/kubelet/pods/9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3/volumes" Nov 03 14:06:21.064428 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:21.064388 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:06:21.064595 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:21.064449 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:06:24.884895 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.884832 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:06:24.961845 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.961806 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-token\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-token\") pod \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " Nov 03 14:06:24.962014 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.961928 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-data\") pod \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " Nov 03 14:06:24.962014 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.961964 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-certs\") pod \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " Nov 03 14:06:24.962118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962039 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-socket\") pod \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " Nov 03 14:06:24.962118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962070 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-envoy\") pod \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " Nov 03 14:06:24.962222 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962123 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-credential-socket\") pod \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " Nov 03 14:06:24.962278 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962217 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-podinfo\") pod \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " Nov 03 14:06:24.962278 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962271 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-kube-api-access-vvr9b\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-kube-api-access-vvr9b\") pod \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " Nov 03 14:06:24.962447 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962377 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istiod-ca-cert\") pod \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\" (UID: \"9a8eb416-4743-4d2d-9b4f-6356ba098cae\") " Nov 03 14:06:24.962511 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962272 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-data" pod "9a8eb416-4743-4d2d-9b4f-6356ba098cae" (UID: "9a8eb416-4743-4d2d-9b4f-6356ba098cae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:24.962511 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962344 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-socket" pod "9a8eb416-4743-4d2d-9b4f-6356ba098cae" (UID: "9a8eb416-4743-4d2d-9b4f-6356ba098cae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:24.962511 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962465 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-credential-socket" pod "9a8eb416-4743-4d2d-9b4f-6356ba098cae" (UID: "9a8eb416-4743-4d2d-9b4f-6356ba098cae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:24.962822 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962685 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-certs" pod "9a8eb416-4743-4d2d-9b4f-6356ba098cae" (UID: "9a8eb416-4743-4d2d-9b4f-6356ba098cae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:24.962924 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.962897 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istiod-ca-cert" pod "9a8eb416-4743-4d2d-9b4f-6356ba098cae" (UID: "9a8eb416-4743-4d2d-9b4f-6356ba098cae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:24.963064 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.963036 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:24.963064 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.963062 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:24.963215 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.963078 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:24.963215 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.963091 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:24.963215 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.963102 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:24.964281 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.964256 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-token" pod "9a8eb416-4743-4d2d-9b4f-6356ba098cae" (UID: "9a8eb416-4743-4d2d-9b4f-6356ba098cae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:24.964489 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.964468 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-kube-api-access-vvr9b" pod "9a8eb416-4743-4d2d-9b4f-6356ba098cae" (UID: "9a8eb416-4743-4d2d-9b4f-6356ba098cae"). InnerVolumeSpecName "kube-api-access-vvr9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:24.964713 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.964690 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-envoy" pod "9a8eb416-4743-4d2d-9b4f-6356ba098cae" (UID: "9a8eb416-4743-4d2d-9b4f-6356ba098cae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:24.964793 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.964748 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-podinfo" pod "9a8eb416-4743-4d2d-9b4f-6356ba098cae" (UID: "9a8eb416-4743-4d2d-9b4f-6356ba098cae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:06:24.982392 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.982366 4541 generic.go:356] "Generic (PLEG): container finished" podID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerID="ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d" exitCode=0 Nov 03 14:06:24.982492 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.982418 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" Nov 03 14:06:24.982492 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.982457 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" event={"ID":"9a8eb416-4743-4d2d-9b4f-6356ba098cae","Type":"ContainerDied","Data":"ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d"} Nov 03 14:06:24.982577 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.982527 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp" event={"ID":"9a8eb416-4743-4d2d-9b4f-6356ba098cae","Type":"ContainerDied","Data":"009a4564deb177550544e36c8c1f908571bfe281a923dafba33d0ebd56694b6a"} Nov 03 14:06:24.982577 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.982565 4541 scope.go:122] "RemoveContainer" containerID="ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d" Nov 03 14:06:24.992940 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:24.992920 4541 scope.go:122] "RemoveContainer" containerID="9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a" Nov 03 14:06:25.001647 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.001631 4541 scope.go:122] "RemoveContainer" containerID="ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d" Nov 03 14:06:25.001932 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:06:25.001911 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d\": container with ID starting with ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d not found: ID does not exist" containerID="ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d" Nov 03 14:06:25.001982 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.001941 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d"} err="failed to get container status \"ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d\": rpc error: code = NotFound desc = could not find container \"ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d\": container with ID starting with ccfc2064d3a15bc4d74e8c742d56868dffb7c2fd831ad2dc018da034cfe5a36d not found: ID does not exist" Nov 03 14:06:25.001982 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.001961 4541 scope.go:122] "RemoveContainer" containerID="9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a" Nov 03 14:06:25.002208 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:06:25.002186 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a\": container with ID starting with 9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a not found: ID does not exist" containerID="9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a" Nov 03 14:06:25.002271 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.002220 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a"} err="failed to get container status \"9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a\": rpc error: code = NotFound desc = could not find container \"9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a\": container with ID starting with 9566f2acce17b27e55d8417df77e1d444f99d59d123988482b144eda9ea9d32a not found: ID does not exist" Nov 03 14:06:25.064306 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.064272 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:25.064306 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.064301 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:25.064306 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.064313 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvr9b\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-kube-api-access-vvr9b\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:25.064547 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.064323 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8eb416-4743-4d2d-9b4f-6356ba098cae-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:25.149885 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.149855 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp"] Nov 03 14:06:25.153487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:25.153459 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-58f896-openshift-default-579fbbd657-tf6zp"] Nov 03 14:06:26.868501 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:06:26.868468 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" path="/var/lib/kubelet/pods/9a8eb416-4743-4d2d-9b4f-6356ba098cae/volumes" Nov 03 14:06:46.892240 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:06:46.892201 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:07:16.890941 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:07:16.890898 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:07:30.753622 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:30.753589 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j"] Nov 03 14:07:35.754070 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:35.754024 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" podUID="b3691312-271b-4ce7-aad6-fc8731f3f62b" containerName="payload-processing" containerID="cri-o://30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5" gracePeriod=25 Nov 03 14:07:36.023706 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.023645 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:07:36.083323 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.083288 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3691312-271b-4ce7-aad6-fc8731f3f62b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3691312-271b-4ce7-aad6-fc8731f3f62b-plugins-config-volume\") pod \"b3691312-271b-4ce7-aad6-fc8731f3f62b\" (UID: \"b3691312-271b-4ce7-aad6-fc8731f3f62b\") " Nov 03 14:07:36.083487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.083359 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3691312-271b-4ce7-aad6-fc8731f3f62b-kube-api-access-dlrzq\" (UniqueName: \"kubernetes.io/projected/b3691312-271b-4ce7-aad6-fc8731f3f62b-kube-api-access-dlrzq\") pod \"b3691312-271b-4ce7-aad6-fc8731f3f62b\" (UID: \"b3691312-271b-4ce7-aad6-fc8731f3f62b\") " Nov 03 14:07:36.083672 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.083645 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3691312-271b-4ce7-aad6-fc8731f3f62b-plugins-config-volume" pod "b3691312-271b-4ce7-aad6-fc8731f3f62b" (UID: "b3691312-271b-4ce7-aad6-fc8731f3f62b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:36.083798 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.083761 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3691312-271b-4ce7-aad6-fc8731f3f62b-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:36.085534 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.085509 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3691312-271b-4ce7-aad6-fc8731f3f62b-kube-api-access-dlrzq" pod "b3691312-271b-4ce7-aad6-fc8731f3f62b" (UID: "b3691312-271b-4ce7-aad6-fc8731f3f62b"). InnerVolumeSpecName "kube-api-access-dlrzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:36.184739 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.184705 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrzq\" (UniqueName: \"kubernetes.io/projected/b3691312-271b-4ce7-aad6-fc8731f3f62b-kube-api-access-dlrzq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:36.436633 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.436586 4541 generic.go:356] "Generic (PLEG): container finished" podID="b3691312-271b-4ce7-aad6-fc8731f3f62b" containerID="30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5" exitCode=0 Nov 03 14:07:36.436848 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.436647 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" Nov 03 14:07:36.436848 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.436656 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" event={"ID":"b3691312-271b-4ce7-aad6-fc8731f3f62b","Type":"ContainerDied","Data":"30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5"} Nov 03 14:07:36.436848 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.436752 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j" event={"ID":"b3691312-271b-4ce7-aad6-fc8731f3f62b","Type":"ContainerDied","Data":"069da0fba35de06c5e1c80d5a640e132a434ef19a3c2a0045dd16070f29b08fb"} Nov 03 14:07:36.436848 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.436792 4541 scope.go:122] "RemoveContainer" containerID="30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5" Nov 03 14:07:36.447405 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.447304 4541 scope.go:122] "RemoveContainer" containerID="30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5" Nov 03 14:07:36.447721 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:07:36.447666 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5\": container with ID starting with 30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5 not found: ID does not exist" containerID="30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5" Nov 03 14:07:36.447721 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.447704 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5"} err="failed to get container status \"30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5\": rpc error: code = NotFound desc = could not find container \"30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5\": container with ID starting with 30f8e10b51ed9412b9d00dfe74512f8cf46057fffce3bfc68751893c816b8cf5 not found: ID does not exist" Nov 03 14:07:36.593862 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.593829 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j"] Nov 03 14:07:36.598784 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.598750 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d5d415-576bcf7ccf-td62j"] Nov 03 14:07:36.868867 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:36.868790 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3691312-271b-4ce7-aad6-fc8731f3f62b" path="/var/lib/kubelet/pods/b3691312-271b-4ce7-aad6-fc8731f3f62b/volumes" Nov 03 14:07:46.891758 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:07:46.891712 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:07:52.753894 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:52.753856 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh"] Nov 03 14:07:52.754338 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:52.754087 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" containerID="cri-o://d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627" gracePeriod=30 Nov 03 14:07:55.310891 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:55.310847 4541 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:07:55.311280 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:55.310931 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:07:57.310954 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:57.310913 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28"] Nov 03 14:07:57.419697 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:57.419654 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798"] Nov 03 14:07:58.037358 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.037331 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:07:58.151869 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.151839 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-kube-api-access-5bscj\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-kube-api-access-5bscj\") pod \"8a34c955-2fbc-4f7b-8a19-7df762af539e\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " Nov 03 14:07:58.152060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.151916 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a34c955-2fbc-4f7b-8a19-7df762af539e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a34c955-2fbc-4f7b-8a19-7df762af539e-istiod-ca-cert\") pod \"8a34c955-2fbc-4f7b-8a19-7df762af539e\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " Nov 03 14:07:58.152060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152008 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-certs\") pod \"8a34c955-2fbc-4f7b-8a19-7df762af539e\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " Nov 03 14:07:58.152060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152045 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-token\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-token\") pod \"8a34c955-2fbc-4f7b-8a19-7df762af539e\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " Nov 03 14:07:58.152242 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152103 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-data\") pod \"8a34c955-2fbc-4f7b-8a19-7df762af539e\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " Nov 03 14:07:58.152242 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152177 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-socket\") pod \"8a34c955-2fbc-4f7b-8a19-7df762af539e\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " Nov 03 14:07:58.152242 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152202 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-envoy\") pod \"8a34c955-2fbc-4f7b-8a19-7df762af539e\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " Nov 03 14:07:58.152467 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152292 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-podinfo\") pod \"8a34c955-2fbc-4f7b-8a19-7df762af539e\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " Nov 03 14:07:58.152467 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152337 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-credential-socket\") pod \"8a34c955-2fbc-4f7b-8a19-7df762af539e\" (UID: \"8a34c955-2fbc-4f7b-8a19-7df762af539e\") " Nov 03 14:07:58.152872 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152334 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a34c955-2fbc-4f7b-8a19-7df762af539e-istiod-ca-cert" pod "8a34c955-2fbc-4f7b-8a19-7df762af539e" (UID: "8a34c955-2fbc-4f7b-8a19-7df762af539e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:58.153077 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152832 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-data" pod "8a34c955-2fbc-4f7b-8a19-7df762af539e" (UID: "8a34c955-2fbc-4f7b-8a19-7df762af539e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.153077 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152943 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-socket" pod "8a34c955-2fbc-4f7b-8a19-7df762af539e" (UID: "8a34c955-2fbc-4f7b-8a19-7df762af539e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.153077 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.152994 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-certs" pod "8a34c955-2fbc-4f7b-8a19-7df762af539e" (UID: "8a34c955-2fbc-4f7b-8a19-7df762af539e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.153321 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.153156 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-credential-socket" pod "8a34c955-2fbc-4f7b-8a19-7df762af539e" (UID: "8a34c955-2fbc-4f7b-8a19-7df762af539e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.156008 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.155982 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-token" pod "8a34c955-2fbc-4f7b-8a19-7df762af539e" (UID: "8a34c955-2fbc-4f7b-8a19-7df762af539e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:58.156008 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.155992 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-envoy" pod "8a34c955-2fbc-4f7b-8a19-7df762af539e" (UID: "8a34c955-2fbc-4f7b-8a19-7df762af539e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:58.156176 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.156000 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-podinfo" pod "8a34c955-2fbc-4f7b-8a19-7df762af539e" (UID: "8a34c955-2fbc-4f7b-8a19-7df762af539e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:07:58.156281 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.156262 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-kube-api-access-5bscj" pod "8a34c955-2fbc-4f7b-8a19-7df762af539e" (UID: "8a34c955-2fbc-4f7b-8a19-7df762af539e"). InnerVolumeSpecName "kube-api-access-5bscj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:58.253752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.253714 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.253752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.253750 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.253752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.253761 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.254005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.253802 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bscj\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-kube-api-access-5bscj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.254005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.253814 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a34c955-2fbc-4f7b-8a19-7df762af539e-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.254005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.253822 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.254005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.253832 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.254005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.253841 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.254005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.253849 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a34c955-2fbc-4f7b-8a19-7df762af539e-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:58.573860 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.573823 4541 generic.go:356] "Generic (PLEG): container finished" podID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerID="d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627" exitCode=0 Nov 03 14:07:58.574328 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.573872 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" Nov 03 14:07:58.574328 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.573911 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" event={"ID":"8a34c955-2fbc-4f7b-8a19-7df762af539e","Type":"ContainerDied","Data":"d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627"} Nov 03 14:07:58.574328 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.573965 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh" event={"ID":"8a34c955-2fbc-4f7b-8a19-7df762af539e","Type":"ContainerDied","Data":"a8552cb7fdcb61fcd68bddf8da0554f0a62fcbd38e28dc5ead4ea5c5d3792db3"} Nov 03 14:07:58.574328 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.573988 4541 scope.go:122] "RemoveContainer" containerID="d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627" Nov 03 14:07:58.585494 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.585473 4541 scope.go:122] "RemoveContainer" containerID="d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627" Nov 03 14:07:58.585784 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:07:58.585752 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627\": container with ID starting with d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627 not found: ID does not exist" containerID="d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627" Nov 03 14:07:58.585864 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.585795 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627"} err="failed to get container status \"d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627\": rpc error: code = NotFound desc = could not find container \"d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627\": container with ID starting with d9954c27a3110038bf9ae1da923eda9e0b3a9359c87bf84e805622cb74149627 not found: ID does not exist" Nov 03 14:07:58.730924 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.730886 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh"] Nov 03 14:07:58.777994 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.777950 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-lg7sh"] Nov 03 14:07:58.868612 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:58.868528 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" path="/var/lib/kubelet/pods/8a34c955-2fbc-4f7b-8a19-7df762af539e/volumes" Nov 03 14:07:59.342141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:07:59.342099 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr"] Nov 03 14:08:02.311270 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.311226 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" podUID="5348e4b3-1960-40c9-9f7b-ff07248f408c" containerName="payload-processing" containerID="cri-o://8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23" gracePeriod=25 Nov 03 14:08:02.420903 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.420851 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" podUID="e748d5bc-eabd-40a8-896e-4da00223a62a" containerName="payload-pre-processing" containerID="cri-o://805af4ae64ba369b2144881fc9c1cf2b95fe6b096095acc564efb5955d9275b0" gracePeriod=25 Nov 03 14:08:02.585381 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.585353 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:08:02.602352 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.602002 4541 generic.go:356] "Generic (PLEG): container finished" podID="5348e4b3-1960-40c9-9f7b-ff07248f408c" containerID="8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23" exitCode=0 Nov 03 14:08:02.602352 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.602103 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" Nov 03 14:08:02.602352 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.602143 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" event={"ID":"5348e4b3-1960-40c9-9f7b-ff07248f408c","Type":"ContainerDied","Data":"8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23"} Nov 03 14:08:02.602352 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.602198 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28" event={"ID":"5348e4b3-1960-40c9-9f7b-ff07248f408c","Type":"ContainerDied","Data":"b55da2ef71ddd5b80333a866ba31324bb47f58ae7a3081c95a5b32dfc397c38e"} Nov 03 14:08:02.602352 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.602230 4541 scope.go:122] "RemoveContainer" containerID="8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23" Nov 03 14:08:02.605470 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.605441 4541 generic.go:356] "Generic (PLEG): container finished" podID="e748d5bc-eabd-40a8-896e-4da00223a62a" containerID="805af4ae64ba369b2144881fc9c1cf2b95fe6b096095acc564efb5955d9275b0" exitCode=0 Nov 03 14:08:02.605571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.605491 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" event={"ID":"e748d5bc-eabd-40a8-896e-4da00223a62a","Type":"ContainerDied","Data":"805af4ae64ba369b2144881fc9c1cf2b95fe6b096095acc564efb5955d9275b0"} Nov 03 14:08:02.617763 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.617728 4541 scope.go:122] "RemoveContainer" containerID="8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23" Nov 03 14:08:02.618117 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:08:02.618091 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23\": container with ID starting with 8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23 not found: ID does not exist" containerID="8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23" Nov 03 14:08:02.618200 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.618124 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23"} err="failed to get container status \"8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23\": rpc error: code = NotFound desc = could not find container \"8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23\": container with ID starting with 8a987b83e143f4664194e2e1cb663601b5018a53c47a98724b1cb4b4461eef23 not found: ID does not exist" Nov 03 14:08:02.693369 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.693339 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:08:02.701548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.701520 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5348e4b3-1960-40c9-9f7b-ff07248f408c-kube-api-access-lmjcz\" (UniqueName: \"kubernetes.io/projected/5348e4b3-1960-40c9-9f7b-ff07248f408c-kube-api-access-lmjcz\") pod \"5348e4b3-1960-40c9-9f7b-ff07248f408c\" (UID: \"5348e4b3-1960-40c9-9f7b-ff07248f408c\") " Nov 03 14:08:02.701676 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.701568 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5348e4b3-1960-40c9-9f7b-ff07248f408c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5348e4b3-1960-40c9-9f7b-ff07248f408c-plugins-config-volume\") pod \"5348e4b3-1960-40c9-9f7b-ff07248f408c\" (UID: \"5348e4b3-1960-40c9-9f7b-ff07248f408c\") " Nov 03 14:08:02.702046 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.702019 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5348e4b3-1960-40c9-9f7b-ff07248f408c-plugins-config-volume" pod "5348e4b3-1960-40c9-9f7b-ff07248f408c" (UID: "5348e4b3-1960-40c9-9f7b-ff07248f408c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:08:02.703645 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.703618 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5348e4b3-1960-40c9-9f7b-ff07248f408c-kube-api-access-lmjcz" pod "5348e4b3-1960-40c9-9f7b-ff07248f408c" (UID: "5348e4b3-1960-40c9-9f7b-ff07248f408c"). InnerVolumeSpecName "kube-api-access-lmjcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:08:02.803100 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.803070 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e748d5bc-eabd-40a8-896e-4da00223a62a-kube-api-access-67v5d\" (UniqueName: \"kubernetes.io/projected/e748d5bc-eabd-40a8-896e-4da00223a62a-kube-api-access-67v5d\") pod \"e748d5bc-eabd-40a8-896e-4da00223a62a\" (UID: \"e748d5bc-eabd-40a8-896e-4da00223a62a\") " Nov 03 14:08:02.803317 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.803138 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e748d5bc-eabd-40a8-896e-4da00223a62a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e748d5bc-eabd-40a8-896e-4da00223a62a-plugins-config-volume\") pod \"e748d5bc-eabd-40a8-896e-4da00223a62a\" (UID: \"e748d5bc-eabd-40a8-896e-4da00223a62a\") " Nov 03 14:08:02.803510 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.803492 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmjcz\" (UniqueName: \"kubernetes.io/projected/5348e4b3-1960-40c9-9f7b-ff07248f408c-kube-api-access-lmjcz\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:02.803572 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.803517 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5348e4b3-1960-40c9-9f7b-ff07248f408c-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:02.803572 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.803514 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e748d5bc-eabd-40a8-896e-4da00223a62a-plugins-config-volume" pod "e748d5bc-eabd-40a8-896e-4da00223a62a" (UID: "e748d5bc-eabd-40a8-896e-4da00223a62a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:08:02.805322 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.805303 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e748d5bc-eabd-40a8-896e-4da00223a62a-kube-api-access-67v5d" pod "e748d5bc-eabd-40a8-896e-4da00223a62a" (UID: "e748d5bc-eabd-40a8-896e-4da00223a62a"). InnerVolumeSpecName "kube-api-access-67v5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:08:02.913829 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.905870 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e748d5bc-eabd-40a8-896e-4da00223a62a-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:02.913829 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:02.905906 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67v5d\" (UniqueName: \"kubernetes.io/projected/e748d5bc-eabd-40a8-896e-4da00223a62a-kube-api-access-67v5d\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:03.013437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:03.013391 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28"] Nov 03 14:08:03.016331 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:03.016301 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-aff1d2-864d5c89d5-nrs28"] Nov 03 14:08:03.614104 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:03.614014 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" event={"ID":"e748d5bc-eabd-40a8-896e-4da00223a62a","Type":"ContainerDied","Data":"4a5c58b2d58102daca18a73e83eef451681b7ef81ee45574a9534ed99f9b28fb"} Nov 03 14:08:03.614581 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:03.614106 4541 scope.go:122] "RemoveContainer" containerID="805af4ae64ba369b2144881fc9c1cf2b95fe6b096095acc564efb5955d9275b0" Nov 03 14:08:03.614581 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:03.614025 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798" Nov 03 14:08:03.816354 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:03.816312 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798"] Nov 03 14:08:03.826979 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:03.826952 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-aff1d2-fcfbf6499-w4798"] Nov 03 14:08:04.343291 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.343247 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" podUID="e825ac61-1ada-4fc7-afb3-ac562d59e4d4" containerName="payload-processing" containerID="cri-o://9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee" gracePeriod=25 Nov 03 14:08:04.543342 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.543307 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7"] Nov 03 14:08:04.543951 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.543930 4541 cpu_manager.go:414] "containerMap: removing container" podUID="9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3" containerName="payload-pre-processing" Nov 03 14:08:04.543951 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.543947 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3" containerName="payload-pre-processing" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.543960 4541 cpu_manager.go:414] "containerMap: removing container" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.543966 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.543985 4541 cpu_manager.go:414] "containerMap: removing container" podUID="5348e4b3-1960-40c9-9f7b-ff07248f408c" containerName="payload-processing" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.543992 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5348e4b3-1960-40c9-9f7b-ff07248f408c" containerName="payload-processing" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544003 4541 cpu_manager.go:414] "containerMap: removing container" podUID="b3691312-271b-4ce7-aad6-fc8731f3f62b" containerName="payload-processing" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544008 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3691312-271b-4ce7-aad6-fc8731f3f62b" containerName="payload-processing" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544022 4541 cpu_manager.go:414] "containerMap: removing container" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544028 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544038 4541 cpu_manager.go:414] "containerMap: removing container" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544042 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544049 4541 cpu_manager.go:414] "containerMap: removing container" podUID="e748d5bc-eabd-40a8-896e-4da00223a62a" containerName="payload-pre-processing" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544056 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e748d5bc-eabd-40a8-896e-4da00223a62a" containerName="payload-pre-processing" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544140 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a1e48d4-0e3a-4fef-b4da-dd28a0f594d3" containerName="payload-pre-processing" Nov 03 14:08:04.544141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544148 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a34c955-2fbc-4f7b-8a19-7df762af539e" containerName="istio-proxy" Nov 03 14:08:04.544911 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544160 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="5348e4b3-1960-40c9-9f7b-ff07248f408c" containerName="payload-processing" Nov 03 14:08:04.544911 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544171 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3691312-271b-4ce7-aad6-fc8731f3f62b" containerName="payload-processing" Nov 03 14:08:04.544911 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544177 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" Nov 03 14:08:04.544911 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.544185 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="e748d5bc-eabd-40a8-896e-4da00223a62a" containerName="payload-pre-processing" Nov 03 14:08:04.547608 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.547584 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.593358 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.593133 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7"] Nov 03 14:08:04.597508 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.597484 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wd2hr\"" Nov 03 14:08:04.620527 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.620501 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:08:04.623351 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.623322 4541 generic.go:356] "Generic (PLEG): container finished" podID="e825ac61-1ada-4fc7-afb3-ac562d59e4d4" containerID="9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee" exitCode=0 Nov 03 14:08:04.623484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.623381 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" Nov 03 14:08:04.623484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.623404 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" event={"ID":"e825ac61-1ada-4fc7-afb3-ac562d59e4d4","Type":"ContainerDied","Data":"9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee"} Nov 03 14:08:04.623484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.623468 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr" event={"ID":"e825ac61-1ada-4fc7-afb3-ac562d59e4d4","Type":"ContainerDied","Data":"5f8da53879502bafca192c86fdc4266c7aad0525a9c9b6148ebad6b6a4695ccd"} Nov 03 14:08:04.623630 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.623501 4541 scope.go:122] "RemoveContainer" containerID="9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee" Nov 03 14:08:04.626118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.626094 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.626232 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.626212 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.626300 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.626281 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6qz\" (UniqueName: \"kubernetes.io/projected/b94a6140-9ce0-476f-9385-a2756e76c1b2-kube-api-access-mb6qz\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.626354 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.626317 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.626429 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.626413 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.626487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.626458 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94a6140-9ce0-476f-9385-a2756e76c1b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.626553 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.626537 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.626621 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.626601 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.626679 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.626663 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.633141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.633116 4541 scope.go:122] "RemoveContainer" containerID="9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee" Nov 03 14:08:04.633405 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:08:04.633383 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee\": container with ID starting with 9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee not found: ID does not exist" containerID="9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee" Nov 03 14:08:04.633490 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.633414 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee"} err="failed to get container status \"9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee\": rpc error: code = NotFound desc = could not find container \"9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee\": container with ID starting with 9d16ac03daea07df916b2af78927485fbf783e871a6b0bf34f99abb290b3f2ee not found: ID does not exist" Nov 03 14:08:04.727799 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.727751 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-plugins-config-volume\") pod \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\" (UID: \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\") " Nov 03 14:08:04.727980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.727866 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-kube-api-access-nhw5g\" (UniqueName: \"kubernetes.io/projected/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-kube-api-access-nhw5g\") pod \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\" (UID: \"e825ac61-1ada-4fc7-afb3-ac562d59e4d4\") " Nov 03 14:08:04.728076 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728055 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728146 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728121 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728211 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728177 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6qz\" (UniqueName: \"kubernetes.io/projected/b94a6140-9ce0-476f-9385-a2756e76c1b2-kube-api-access-mb6qz\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728206 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728216 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-plugins-config-volume" pod "e825ac61-1ada-4fc7-afb3-ac562d59e4d4" (UID: "e825ac61-1ada-4fc7-afb3-ac562d59e4d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:08:04.728269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728232 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728268 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94a6140-9ce0-476f-9385-a2756e76c1b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728312 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728353 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728405 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728660 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728478 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:04.728660 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728479 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.728660 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.728537 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.729068 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.729019 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.729185 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.729109 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.729425 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.729404 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b94a6140-9ce0-476f-9385-a2756e76c1b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.730287 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.730253 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-kube-api-access-nhw5g" pod "e825ac61-1ada-4fc7-afb3-ac562d59e4d4" (UID: "e825ac61-1ada-4fc7-afb3-ac562d59e4d4"). InnerVolumeSpecName "kube-api-access-nhw5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:08:04.730688 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.730669 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.731198 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.731173 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.783138 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.783099 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b94a6140-9ce0-476f-9385-a2756e76c1b2-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.783291 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.783138 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6qz\" (UniqueName: \"kubernetes.io/projected/b94a6140-9ce0-476f-9385-a2756e76c1b2-kube-api-access-mb6qz\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-m7hd7\" (UID: \"b94a6140-9ce0-476f-9385-a2756e76c1b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.829730 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.829692 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhw5g\" (UniqueName: \"kubernetes.io/projected/e825ac61-1ada-4fc7-afb3-ac562d59e4d4-kube-api-access-nhw5g\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:04.864620 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.864528 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:04.870419 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.870392 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5348e4b3-1960-40c9-9f7b-ff07248f408c" path="/var/lib/kubelet/pods/5348e4b3-1960-40c9-9f7b-ff07248f408c/volumes" Nov 03 14:08:04.870885 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:04.870862 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e748d5bc-eabd-40a8-896e-4da00223a62a" path="/var/lib/kubelet/pods/e748d5bc-eabd-40a8-896e-4da00223a62a/volumes" Nov 03 14:08:05.035851 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.035818 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr"] Nov 03 14:08:05.038664 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.038633 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-966a03-85d9fff-ktzrr"] Nov 03 14:08:05.285116 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.285079 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7"] Nov 03 14:08:05.289836 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:08:05.289806 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94a6140_9ce0_476f_9385_a2756e76c1b2.slice/crio-875b1e06cd69cbf3438dd5e2dbd8de46e596d06efd5ad97492e266b3d2a0104c WatchSource:0}: Error finding container 875b1e06cd69cbf3438dd5e2dbd8de46e596d06efd5ad97492e266b3d2a0104c: Status 404 returned error can't find the container with id 875b1e06cd69cbf3438dd5e2dbd8de46e596d06efd5ad97492e266b3d2a0104c Nov 03 14:08:05.292454 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.292421 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:08:05.292564 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.292506 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:08:05.292564 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.292551 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:08:05.634196 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.634115 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" event={"ID":"b94a6140-9ce0-476f-9385-a2756e76c1b2","Type":"ContainerStarted","Data":"52069b8ec4b50646375c4441fcd40e258b90065a7679f823bc3b6a9b7c0c6de4"} Nov 03 14:08:05.634196 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.634167 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" event={"ID":"b94a6140-9ce0-476f-9385-a2756e76c1b2","Type":"ContainerStarted","Data":"875b1e06cd69cbf3438dd5e2dbd8de46e596d06efd5ad97492e266b3d2a0104c"} Nov 03 14:08:05.750721 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.750659 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podStartSLOduration=1.750645532 podStartE2EDuration="1.750645532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:08:05.74833597 +0000 UTC m=+3817.062307830" watchObservedRunningTime="2026-11-03 14:08:05.750645532 +0000 UTC m=+3817.064617391" Nov 03 14:08:05.865043 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:05.864970 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:06.867169 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:06.867134 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 14:08:06.867564 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:06.867190 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podUID="b94a6140-9ce0-476f-9385-a2756e76c1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 14:08:06.870687 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:06.870660 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e825ac61-1ada-4fc7-afb3-ac562d59e4d4" path="/var/lib/kubelet/pods/e825ac61-1ada-4fc7-afb3-ac562d59e4d4/volumes" Nov 03 14:08:07.864907 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:07.864871 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 14:08:07.865095 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:07.864931 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podUID="b94a6140-9ce0-476f-9385-a2756e76c1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 14:08:08.866196 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:08.866159 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 14:08:08.866620 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:08.866223 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podUID="b94a6140-9ce0-476f-9385-a2756e76c1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 14:08:09.867793 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:09.865218 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 14:08:09.867793 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:09.865282 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podUID="b94a6140-9ce0-476f-9385-a2756e76c1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 14:08:10.865235 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:10.865199 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 14:08:10.865421 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:10.865254 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podUID="b94a6140-9ce0-476f-9385-a2756e76c1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 14:08:11.865729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:11.865685 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 14:08:11.866275 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:11.865747 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podUID="b94a6140-9ce0-476f-9385-a2756e76c1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 14:08:12.864947 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:12.864908 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 14:08:12.865120 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:12.864989 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podUID="b94a6140-9ce0-476f-9385-a2756e76c1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 14:08:13.922394 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:13.922357 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:13.922910 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:13.922845 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:13.923799 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:13.923749 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" Nov 03 14:08:16.889311 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:08:16.889264 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:08:19.655744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.655707 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf"] Nov 03 14:08:19.656287 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.656272 4541 cpu_manager.go:414] "containerMap: removing container" podUID="e825ac61-1ada-4fc7-afb3-ac562d59e4d4" containerName="payload-processing" Nov 03 14:08:19.656331 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.656287 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e825ac61-1ada-4fc7-afb3-ac562d59e4d4" containerName="payload-processing" Nov 03 14:08:19.656464 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.656453 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="e825ac61-1ada-4fc7-afb3-ac562d59e4d4" containerName="payload-processing" Nov 03 14:08:19.656520 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.656467 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8eb416-4743-4d2d-9b4f-6356ba098cae" containerName="istio-proxy" Nov 03 14:08:19.661042 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.661022 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.736335 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.736280 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf"] Nov 03 14:08:19.835354 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.835321 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.835354 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.835358 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303b207-db31-4672-aea3-cb93766856db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.835615 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.835484 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.835615 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.835511 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303b207-db31-4672-aea3-cb93766856db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.835615 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.835606 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.835724 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.835639 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.835724 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.835702 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmzk\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-kube-api-access-dmmzk\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.835840 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.835751 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.835840 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.835803 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.937371 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937284 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmzk\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-kube-api-access-dmmzk\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.937371 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937337 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.937371 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937364 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.937664 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937453 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.937664 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937479 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303b207-db31-4672-aea3-cb93766856db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.937664 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937608 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.937664 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937631 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303b207-db31-4672-aea3-cb93766856db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.937900 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937706 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.937900 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937740 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.938008 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.937956 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.938118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.938095 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.938579 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.938243 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.939759 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.938503 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.939881 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.939145 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303b207-db31-4672-aea3-cb93766856db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.942223 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.942194 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303b207-db31-4672-aea3-cb93766856db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.942358 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.941624 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.992446 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.992416 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:19.992446 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:19.992442 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmzk\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-kube-api-access-dmmzk\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-b5gdf\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:20.077897 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.077864 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw"] Nov 03 14:08:20.083301 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.083280 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:20.133708 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.133664 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f11801-dockercfg-t945f\"" Nov 03 14:08:20.133708 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.133693 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f11801\"" Nov 03 14:08:20.138014 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.137984 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw"] Nov 03 14:08:20.241363 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.241283 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw\" (UID: \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:20.241522 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.241442 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjcvz\" (UniqueName: \"kubernetes.io/projected/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-kube-api-access-vjcvz\") pod \"payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw\" (UID: \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:20.275391 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.275360 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:20.344268 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.344229 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjcvz\" (UniqueName: \"kubernetes.io/projected/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-kube-api-access-vjcvz\") pod \"payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw\" (UID: \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:20.344453 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.344429 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw\" (UID: \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:20.345559 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.345395 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw\" (UID: \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:20.399723 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.399697 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjcvz\" (UniqueName: \"kubernetes.io/projected/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-kube-api-access-vjcvz\") pod \"payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw\" (UID: \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:20.698006 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.697972 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:20.699144 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.699121 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf"] Nov 03 14:08:20.700680 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:08:20.700655 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9303b207_db31_4672_aea3_cb93766856db.slice/crio-73c32be8770621810f8c8931acfa3a1382c4d0b6c135f897a578f807adefa3c9 WatchSource:0}: Error finding container 73c32be8770621810f8c8931acfa3a1382c4d0b6c135f897a578f807adefa3c9: Status 404 returned error can't find the container with id 73c32be8770621810f8c8931acfa3a1382c4d0b6c135f897a578f807adefa3c9 Nov 03 14:08:20.703237 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.703201 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:08:20.703327 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.703268 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:08:20.703327 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.703298 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:08:20.751609 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:20.751574 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" event={"ID":"9303b207-db31-4672-aea3-cb93766856db","Type":"ContainerStarted","Data":"73c32be8770621810f8c8931acfa3a1382c4d0b6c135f897a578f807adefa3c9"} Nov 03 14:08:21.077857 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:21.077820 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw"] Nov 03 14:08:21.079146 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:08:21.079119 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9ac7ff_c079_4a31_9a4c_7d4f23095492.slice/crio-e0b85268020d5ea8a1c0e50dd32367a9eb1f147ebce3cfeb7f88703409e04e3b WatchSource:0}: Error finding container e0b85268020d5ea8a1c0e50dd32367a9eb1f147ebce3cfeb7f88703409e04e3b: Status 404 returned error can't find the container with id e0b85268020d5ea8a1c0e50dd32367a9eb1f147ebce3cfeb7f88703409e04e3b Nov 03 14:08:21.760549 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:21.760196 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" event={"ID":"9303b207-db31-4672-aea3-cb93766856db","Type":"ContainerStarted","Data":"cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c"} Nov 03 14:08:21.762064 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:21.762036 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" event={"ID":"4e9ac7ff-c079-4a31-9a4c-7d4f23095492","Type":"ContainerStarted","Data":"2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef"} Nov 03 14:08:21.762216 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:21.762087 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" event={"ID":"4e9ac7ff-c079-4a31-9a4c-7d4f23095492","Type":"ContainerStarted","Data":"e0b85268020d5ea8a1c0e50dd32367a9eb1f147ebce3cfeb7f88703409e04e3b"} Nov 03 14:08:21.762216 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:21.762116 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:21.922040 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:21.921987 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podStartSLOduration=2.921972986 podStartE2EDuration="2.921972986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:08:21.920844502 +0000 UTC m=+3833.234816362" watchObservedRunningTime="2026-11-03 14:08:21.921972986 +0000 UTC m=+3833.235944845" Nov 03 14:08:22.033431 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:22.033313 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" podStartSLOduration=2.033296046 podStartE2EDuration="2.033296046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:08:22.030719113 +0000 UTC m=+3833.344690974" watchObservedRunningTime="2026-11-03 14:08:22.033296046 +0000 UTC m=+3833.347267905" Nov 03 14:08:22.276404 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:22.276364 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:22.278018 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:22.277990 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 14:08:22.278177 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:22.278043 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 14:08:23.276452 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:23.276410 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 14:08:23.276958 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:23.276466 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 14:08:24.276506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:24.276464 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 14:08:24.276954 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:24.276524 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 14:08:25.276050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:25.276013 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 14:08:25.276264 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:25.276061 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 14:08:26.276295 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:26.276254 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 14:08:26.276702 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:26.276321 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 14:08:27.276628 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:27.276584 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 14:08:27.277157 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:27.276642 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 14:08:28.276409 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:28.276360 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 03 14:08:28.276607 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:28.276432 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 03 14:08:29.925276 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:29.925235 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:29.925912 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:29.925884 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:29.926680 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:29.926664 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:08:32.771613 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:08:32.771578 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:08:46.890496 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:08:46.890457 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:09:05.042494 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.042456 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b"] Nov 03 14:09:05.046959 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.046935 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.182421 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.182382 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.182603 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.182436 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.182660 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.182626 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.182713 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.182672 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.182804 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.182786 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.182870 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.182832 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.182919 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.182865 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.182982 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.182967 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.183081 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.183064 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55kj\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-kube-api-access-n55kj\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.245890 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.245858 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b"] Nov 03 14:09:05.284312 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284277 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.284312 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284317 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.284635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284363 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.284635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284396 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.284635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284415 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.284635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284453 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.284635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284516 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55kj\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-kube-api-access-n55kj\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.284635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284564 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.284635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284584 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.285037 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284803 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.285037 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.284821 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.285158 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.285072 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.285324 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.285299 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.285485 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.285451 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.287137 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.287115 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.287566 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.287549 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.339380 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.339299 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.339514 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.339414 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55kj\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-kube-api-access-n55kj\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9gj2b\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.360298 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.360270 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:05.780467 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.780434 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b"] Nov 03 14:09:05.781627 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:09:05.781599 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e55ba40_fb4f_4a14_8d11_8226803aa43c.slice/crio-e8395799b2c4ce917bee6074af57824e760c23bae0207e51dd74ebf54c8f7876 WatchSource:0}: Error finding container e8395799b2c4ce917bee6074af57824e760c23bae0207e51dd74ebf54c8f7876: Status 404 returned error can't find the container with id e8395799b2c4ce917bee6074af57824e760c23bae0207e51dd74ebf54c8f7876 Nov 03 14:09:05.784268 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.784235 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:09:05.784348 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.784316 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:09:05.784389 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:05.784356 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:09:06.061898 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:06.061809 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" event={"ID":"4e55ba40-fb4f-4a14-8d11-8226803aa43c","Type":"ContainerStarted","Data":"6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf"} Nov 03 14:09:06.061898 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:06.061879 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" event={"ID":"4e55ba40-fb4f-4a14-8d11-8226803aa43c","Type":"ContainerStarted","Data":"e8395799b2c4ce917bee6074af57824e760c23bae0207e51dd74ebf54c8f7876"} Nov 03 14:09:06.209070 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:06.209021 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podStartSLOduration=2.20900769 podStartE2EDuration="2.20900769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:09:06.208093553 +0000 UTC m=+3877.522065413" watchObservedRunningTime="2026-11-03 14:09:06.20900769 +0000 UTC m=+3877.522979551" Nov 03 14:09:06.360935 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:06.360843 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:06.362614 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:06.362579 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 14:09:06.362759 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:06.362644 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 14:09:07.361170 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:07.361125 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 14:09:07.361680 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:07.361183 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 14:09:08.361308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:08.361272 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 14:09:08.361711 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:08.361330 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 14:09:09.360890 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:09.360851 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 14:09:09.361096 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:09.360920 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 14:09:10.361175 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:10.361128 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 14:09:10.361695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:10.361220 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 14:09:11.360854 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:11.360755 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 14:09:11.361069 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:11.360878 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 14:09:12.361074 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:12.361031 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 14:09:12.361500 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:12.361095 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 14:09:13.361302 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:13.361262 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 03 14:09:13.361803 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:13.361334 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 03 14:09:14.424949 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:14.424915 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:14.426265 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:14.426230 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:14.427019 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:14.426996 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:09:16.890874 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:09:16.890833 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:09:25.705263 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:25.705225 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg"] Nov 03 14:09:25.705744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:25.705501 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" containerID="cri-o://69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406" gracePeriod=30 Nov 03 14:09:29.362329 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:29.362299 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:09:29.369586 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:29.369566 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:09:30.962220 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:30.962186 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:09:31.140127 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.140027 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-token\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-token\") pod \"6794d684-2185-47cd-9c22-656bf45d1a4c\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " Nov 03 14:09:31.140127 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.140105 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-kube-api-access-sb5gr\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-kube-api-access-sb5gr\") pod \"6794d684-2185-47cd-9c22-656bf45d1a4c\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " Nov 03 14:09:31.140389 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.140183 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-credential-socket\") pod \"6794d684-2185-47cd-9c22-656bf45d1a4c\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " Nov 03 14:09:31.140389 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.140227 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-data\") pod \"6794d684-2185-47cd-9c22-656bf45d1a4c\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " Nov 03 14:09:31.140389 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.140275 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-envoy\") pod \"6794d684-2185-47cd-9c22-656bf45d1a4c\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " Nov 03 14:09:31.140389 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.140307 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6794d684-2185-47cd-9c22-656bf45d1a4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794d684-2185-47cd-9c22-656bf45d1a4c-istiod-ca-cert\") pod \"6794d684-2185-47cd-9c22-656bf45d1a4c\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " Nov 03 14:09:31.140610 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.140429 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-podinfo\") pod \"6794d684-2185-47cd-9c22-656bf45d1a4c\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " Nov 03 14:09:31.140610 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.140469 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-certs\") pod \"6794d684-2185-47cd-9c22-656bf45d1a4c\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " Nov 03 14:09:31.140610 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.140557 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-socket\") pod \"6794d684-2185-47cd-9c22-656bf45d1a4c\" (UID: \"6794d684-2185-47cd-9c22-656bf45d1a4c\") " Nov 03 14:09:31.141071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.141043 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-data" pod "6794d684-2185-47cd-9c22-656bf45d1a4c" (UID: "6794d684-2185-47cd-9c22-656bf45d1a4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:09:31.141309 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.141281 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-socket" pod "6794d684-2185-47cd-9c22-656bf45d1a4c" (UID: "6794d684-2185-47cd-9c22-656bf45d1a4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:09:31.141484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.141397 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-certs" pod "6794d684-2185-47cd-9c22-656bf45d1a4c" (UID: "6794d684-2185-47cd-9c22-656bf45d1a4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:09:31.141683 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.141661 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6794d684-2185-47cd-9c22-656bf45d1a4c-istiod-ca-cert" pod "6794d684-2185-47cd-9c22-656bf45d1a4c" (UID: "6794d684-2185-47cd-9c22-656bf45d1a4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:09:31.142181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.142161 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-credential-socket" pod "6794d684-2185-47cd-9c22-656bf45d1a4c" (UID: "6794d684-2185-47cd-9c22-656bf45d1a4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:09:31.143508 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.143260 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-envoy" pod "6794d684-2185-47cd-9c22-656bf45d1a4c" (UID: "6794d684-2185-47cd-9c22-656bf45d1a4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:09:31.143508 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.143310 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-kube-api-access-sb5gr" pod "6794d684-2185-47cd-9c22-656bf45d1a4c" (UID: "6794d684-2185-47cd-9c22-656bf45d1a4c"). InnerVolumeSpecName "kube-api-access-sb5gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:09:31.143813 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.143755 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-token" pod "6794d684-2185-47cd-9c22-656bf45d1a4c" (UID: "6794d684-2185-47cd-9c22-656bf45d1a4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:09:31.143936 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.143912 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-podinfo" pod "6794d684-2185-47cd-9c22-656bf45d1a4c" (UID: "6794d684-2185-47cd-9c22-656bf45d1a4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:09:31.217828 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.217788 4541 generic.go:356] "Generic (PLEG): container finished" podID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerID="69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406" exitCode=0 Nov 03 14:09:31.218023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.217838 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" Nov 03 14:09:31.218023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.217863 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" event={"ID":"6794d684-2185-47cd-9c22-656bf45d1a4c","Type":"ContainerDied","Data":"69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406"} Nov 03 14:09:31.218023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.217933 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg" event={"ID":"6794d684-2185-47cd-9c22-656bf45d1a4c","Type":"ContainerDied","Data":"87ba34cfb7eb39731cfb8c37a51dda0aaac58c43b3dc8f82da9da0aeb042b892"} Nov 03 14:09:31.218023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.217963 4541 scope.go:122] "RemoveContainer" containerID="69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406" Nov 03 14:09:31.228619 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.228588 4541 scope.go:122] "RemoveContainer" containerID="8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7" Nov 03 14:09:31.237589 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.237569 4541 scope.go:122] "RemoveContainer" containerID="69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406" Nov 03 14:09:31.237858 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:09:31.237834 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406\": container with ID starting with 69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406 not found: ID does not exist" containerID="69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406" Nov 03 14:09:31.237968 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.237868 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406"} err="failed to get container status \"69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406\": rpc error: code = NotFound desc = could not find container \"69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406\": container with ID starting with 69ef3301b485e2f24da54d57640e22a9dfb607846e5d19a32516d36a3ac64406 not found: ID does not exist" Nov 03 14:09:31.237968 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.237892 4541 scope.go:122] "RemoveContainer" containerID="8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7" Nov 03 14:09:31.238167 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:09:31.238150 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7\": container with ID starting with 8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7 not found: ID does not exist" containerID="8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7" Nov 03 14:09:31.238206 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.238182 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7"} err="failed to get container status \"8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7\": rpc error: code = NotFound desc = could not find container \"8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7\": container with ID starting with 8e20034d669939eaa8f7f9bceafc94206b467147982c5fb149883f16031d5ae7 not found: ID does not exist" Nov 03 14:09:31.241873 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.241849 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:31.241937 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.241876 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:31.241937 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.241887 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb5gr\" (UniqueName: \"kubernetes.io/projected/6794d684-2185-47cd-9c22-656bf45d1a4c-kube-api-access-sb5gr\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:31.241937 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.241898 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:31.241937 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.241906 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:31.241937 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.241915 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:31.241937 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.241924 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6794d684-2185-47cd-9c22-656bf45d1a4c-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:31.241937 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.241937 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6794d684-2185-47cd-9c22-656bf45d1a4c-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:31.242149 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.241948 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6794d684-2185-47cd-9c22-656bf45d1a4c-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:31.376074 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.376040 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg"] Nov 03 14:09:31.380394 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:31.380367 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-966a03-openshift-default-76d686c88f-7bnlg"] Nov 03 14:09:32.875252 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:32.875215 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" path="/var/lib/kubelet/pods/6794d684-2185-47cd-9c22-656bf45d1a4c/volumes" Nov 03 14:09:34.413358 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.413321 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf"] Nov 03 14:09:34.413913 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.413898 4541 cpu_manager.go:414] "containerMap: removing container" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" Nov 03 14:09:34.413964 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.413914 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" Nov 03 14:09:34.413964 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.413935 4541 cpu_manager.go:414] "containerMap: removing container" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" Nov 03 14:09:34.413964 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.413940 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" Nov 03 14:09:34.414062 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.414030 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" Nov 03 14:09:34.414062 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.414049 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="6794d684-2185-47cd-9c22-656bf45d1a4c" containerName="istio-proxy" Nov 03 14:09:34.419136 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.419116 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.468618 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.468590 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-f11801-serving-cert\"" Nov 03 14:09:34.518550 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.518518 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf"] Nov 03 14:09:34.580552 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.580514 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98513768-8784-4b97-8e08-80396a97c28a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.580741 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.580714 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phrx7\" (UniqueName: \"kubernetes.io/projected/98513768-8784-4b97-8e08-80396a97c28a-kube-api-access-phrx7\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.580976 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.580956 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.581048 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.580992 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-metrics-certs\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.682646 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.682545 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.682646 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.682592 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-metrics-certs\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.682909 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.682653 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98513768-8784-4b97-8e08-80396a97c28a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.682909 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:09:34.682729 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-serving-cert: secret "maas-api-e2e-extmod-a-f11801-serving-cert" not found Nov 03 14:09:34.682909 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.682766 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phrx7\" (UniqueName: \"kubernetes.io/projected/98513768-8784-4b97-8e08-80396a97c28a-kube-api-access-phrx7\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.682909 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:09:34.682843 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls podName:98513768-8784-4b97-8e08-80396a97c28a nodeName:}" failed. No retries permitted until 2026-11-03 14:09:35.182818629 +0000 UTC m=+3906.496790491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls") pod "maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" (UID: "98513768-8784-4b97-8e08-80396a97c28a") : secret "maas-api-e2e-extmod-a-f11801-serving-cert" not found Nov 03 14:09:34.683598 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.683572 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98513768-8784-4b97-8e08-80396a97c28a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.685373 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.685349 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-metrics-certs\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:34.738795 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:34.738747 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phrx7\" (UniqueName: \"kubernetes.io/projected/98513768-8784-4b97-8e08-80396a97c28a-kube-api-access-phrx7\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:35.188663 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:35.188612 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:35.191253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:35.191229 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:35.337223 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:35.337186 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:35.711699 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:35.711665 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf"] Nov 03 14:09:35.714443 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:35.714413 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:09:36.252313 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:36.252275 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" event={"ID":"98513768-8784-4b97-8e08-80396a97c28a","Type":"ContainerStarted","Data":"994e8750d56c053be3d415ef5a90ecbf62039894230c480b6dc73dc7dfa5a3de"} Nov 03 14:09:37.259826 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:37.259792 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" event={"ID":"98513768-8784-4b97-8e08-80396a97c28a","Type":"ContainerStarted","Data":"a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba"} Nov 03 14:09:37.260229 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:37.259922 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:37.397169 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:37.397104 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" podStartSLOduration=2.469021869 podStartE2EDuration="3.397086177s" totalImagesPullingTime="928.064308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:09:37.394981551 +0000 UTC m=+3908.708953393" watchObservedRunningTime="2026-11-03 14:09:37.397086177 +0000 UTC m=+3908.711058039" Nov 03 14:09:38.796593 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.796552 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f"] Nov 03 14:09:38.801638 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.801611 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:09:38.824594 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.824562 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f\" (UID: \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:09:38.824748 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.824611 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th79w\" (UniqueName: \"kubernetes.io/projected/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-kube-api-access-th79w\") pod \"payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f\" (UID: \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:09:38.854144 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.854109 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f"] Nov 03 14:09:38.896613 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.896584 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6375e7-dockercfg-vt59p\"" Nov 03 14:09:38.896613 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.896584 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6375e7\"" Nov 03 14:09:38.925881 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.925837 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f\" (UID: \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:09:38.926076 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.925904 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th79w\" (UniqueName: \"kubernetes.io/projected/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-kube-api-access-th79w\") pod \"payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f\" (UID: \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:09:38.926630 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.926609 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f\" (UID: \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:09:38.980460 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:38.980426 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-th79w\" (UniqueName: \"kubernetes.io/projected/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-kube-api-access-th79w\") pod \"payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f\" (UID: \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:09:39.118036 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:39.117945 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:09:39.490187 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:39.490121 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f"] Nov 03 14:09:39.491034 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:09:39.491007 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc7c5a6_f412_41c3_b0d8_8e75ff424e1d.slice/crio-acab883e273f2fd0f2622f83551df7fdc54c76769e7f7f7fc6bdf8dcf6ffa0c9 WatchSource:0}: Error finding container acab883e273f2fd0f2622f83551df7fdc54c76769e7f7f7fc6bdf8dcf6ffa0c9: Status 404 returned error can't find the container with id acab883e273f2fd0f2622f83551df7fdc54c76769e7f7f7fc6bdf8dcf6ffa0c9 Nov 03 14:09:40.278710 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:40.278616 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" event={"ID":"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d","Type":"ContainerStarted","Data":"bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf"} Nov 03 14:09:40.278710 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:40.278675 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" event={"ID":"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d","Type":"ContainerStarted","Data":"acab883e273f2fd0f2622f83551df7fdc54c76769e7f7f7fc6bdf8dcf6ffa0c9"} Nov 03 14:09:40.279158 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:40.278823 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:09:40.387987 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:40.387925 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" podStartSLOduration=2.387910814 podStartE2EDuration="2.387910814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:09:40.38686943 +0000 UTC m=+3911.700841291" watchObservedRunningTime="2026-11-03 14:09:40.387910814 +0000 UTC m=+3911.701882708" Nov 03 14:09:43.271555 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:43.271523 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:09:46.893834 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:09:46.893763 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:09:51.288765 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:09:51.288734 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:10:16.890760 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:10:16.890653 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:10:32.594435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.594394 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk"] Nov 03 14:10:32.603839 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.603795 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:32.653571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.653531 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk"] Nov 03 14:10:32.681658 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.681615 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92466625-51ae-43a0-b3a9-3a035ac33f93-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk\" (UID: \"92466625-51ae-43a0-b3a9-3a035ac33f93\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:32.681872 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.681764 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppb7x\" (UniqueName: \"kubernetes.io/projected/92466625-51ae-43a0-b3a9-3a035ac33f93-kube-api-access-ppb7x\") pod \"payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk\" (UID: \"92466625-51ae-43a0-b3a9-3a035ac33f93\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:32.697131 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.697082 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b6c8c7\"" Nov 03 14:10:32.697677 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.697653 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b6c8c7-dockercfg-49jwq\"" Nov 03 14:10:32.736296 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.736262 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg"] Nov 03 14:10:32.740692 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.740669 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:32.783301 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.783267 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppb7x\" (UniqueName: \"kubernetes.io/projected/92466625-51ae-43a0-b3a9-3a035ac33f93-kube-api-access-ppb7x\") pod \"payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk\" (UID: \"92466625-51ae-43a0-b3a9-3a035ac33f93\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:32.783505 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.783328 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5mh\" (UniqueName: \"kubernetes.io/projected/8c443546-6048-42d4-8966-d6cfe7242c2b-kube-api-access-rc5mh\") pod \"payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg\" (UID: \"8c443546-6048-42d4-8966-d6cfe7242c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:32.783505 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.783397 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c443546-6048-42d4-8966-d6cfe7242c2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg\" (UID: \"8c443546-6048-42d4-8966-d6cfe7242c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:32.783505 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.783417 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92466625-51ae-43a0-b3a9-3a035ac33f93-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk\" (UID: \"92466625-51ae-43a0-b3a9-3a035ac33f93\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:32.784085 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.784065 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92466625-51ae-43a0-b3a9-3a035ac33f93-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk\" (UID: \"92466625-51ae-43a0-b3a9-3a035ac33f93\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:32.794327 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.794292 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg"] Nov 03 14:10:32.837870 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.837836 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppb7x\" (UniqueName: \"kubernetes.io/projected/92466625-51ae-43a0-b3a9-3a035ac33f93-kube-api-access-ppb7x\") pod \"payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk\" (UID: \"92466625-51ae-43a0-b3a9-3a035ac33f93\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:32.884653 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.884572 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c443546-6048-42d4-8966-d6cfe7242c2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg\" (UID: \"8c443546-6048-42d4-8966-d6cfe7242c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:32.884835 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.884728 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5mh\" (UniqueName: \"kubernetes.io/projected/8c443546-6048-42d4-8966-d6cfe7242c2b-kube-api-access-rc5mh\") pod \"payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg\" (UID: \"8c443546-6048-42d4-8966-d6cfe7242c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:32.885315 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.885297 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c443546-6048-42d4-8966-d6cfe7242c2b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg\" (UID: \"8c443546-6048-42d4-8966-d6cfe7242c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:32.918958 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.918905 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:32.939918 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:32.939890 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5mh\" (UniqueName: \"kubernetes.io/projected/8c443546-6048-42d4-8966-d6cfe7242c2b-kube-api-access-rc5mh\") pod \"payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg\" (UID: \"8c443546-6048-42d4-8966-d6cfe7242c2b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:33.054145 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.054109 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:33.289626 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.289591 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk"] Nov 03 14:10:33.291240 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:10:33.291211 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92466625_51ae_43a0_b3a9_3a035ac33f93.slice/crio-e7d0925d14cbd2f613dd81f795488db70cc72f2befb5db6862099b923e0a46d5 WatchSource:0}: Error finding container e7d0925d14cbd2f613dd81f795488db70cc72f2befb5db6862099b923e0a46d5: Status 404 returned error can't find the container with id e7d0925d14cbd2f613dd81f795488db70cc72f2befb5db6862099b923e0a46d5 Nov 03 14:10:33.437100 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.437069 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg"] Nov 03 14:10:33.438190 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:10:33.438165 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c443546_6048_42d4_8966_d6cfe7242c2b.slice/crio-d8bf8944db3e31dbd9a19ab1ab549a5d11b6af786fe8cad60dbd0b35054191c4 WatchSource:0}: Error finding container d8bf8944db3e31dbd9a19ab1ab549a5d11b6af786fe8cad60dbd0b35054191c4: Status 404 returned error can't find the container with id d8bf8944db3e31dbd9a19ab1ab549a5d11b6af786fe8cad60dbd0b35054191c4 Nov 03 14:10:33.616195 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.616096 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" event={"ID":"8c443546-6048-42d4-8966-d6cfe7242c2b","Type":"ContainerStarted","Data":"b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750"} Nov 03 14:10:33.616195 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.616166 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:33.616195 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.616193 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" event={"ID":"8c443546-6048-42d4-8966-d6cfe7242c2b","Type":"ContainerStarted","Data":"d8bf8944db3e31dbd9a19ab1ab549a5d11b6af786fe8cad60dbd0b35054191c4"} Nov 03 14:10:33.617767 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.617741 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" event={"ID":"92466625-51ae-43a0-b3a9-3a035ac33f93","Type":"ContainerStarted","Data":"57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb"} Nov 03 14:10:33.617904 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.617809 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:33.617904 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.617832 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" event={"ID":"92466625-51ae-43a0-b3a9-3a035ac33f93","Type":"ContainerStarted","Data":"e7d0925d14cbd2f613dd81f795488db70cc72f2befb5db6862099b923e0a46d5"} Nov 03 14:10:33.728569 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.728517 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" podStartSLOduration=1.728503289 podStartE2EDuration="1.728503289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:33.728124871 +0000 UTC m=+3965.042096730" watchObservedRunningTime="2026-11-03 14:10:33.728503289 +0000 UTC m=+3965.042475150" Nov 03 14:10:33.842925 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:33.842863 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" podStartSLOduration=1.842846953 podStartE2EDuration="1.842846953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:10:33.839017504 +0000 UTC m=+3965.152989364" watchObservedRunningTime="2026-11-03 14:10:33.842846953 +0000 UTC m=+3965.156818813" Nov 03 14:10:44.628057 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:44.628023 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:10:44.628475 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:10:44.628097 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:10:46.891119 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:10:46.891081 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:11:16.891250 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:11:16.891158 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:11:22.156668 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:22.156618 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw"] Nov 03 14:11:27.156992 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.156944 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" podUID="4e9ac7ff-c079-4a31-9a4c-7d4f23095492" containerName="payload-pre-processing" containerID="cri-o://2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef" gracePeriod=25 Nov 03 14:11:27.448018 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.447991 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:11:27.543585 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.543555 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-plugins-config-volume\") pod \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\" (UID: \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\") " Nov 03 14:11:27.543821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.543618 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-kube-api-access-vjcvz\" (UniqueName: \"kubernetes.io/projected/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-kube-api-access-vjcvz\") pod \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\" (UID: \"4e9ac7ff-c079-4a31-9a4c-7d4f23095492\") " Nov 03 14:11:27.544031 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.544008 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-plugins-config-volume" pod "4e9ac7ff-c079-4a31-9a4c-7d4f23095492" (UID: "4e9ac7ff-c079-4a31-9a4c-7d4f23095492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:11:27.544145 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.544127 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:11:27.545852 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.545829 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-kube-api-access-vjcvz" pod "4e9ac7ff-c079-4a31-9a4c-7d4f23095492" (UID: "4e9ac7ff-c079-4a31-9a4c-7d4f23095492"). InnerVolumeSpecName "kube-api-access-vjcvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:11:27.645629 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.645589 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjcvz\" (UniqueName: \"kubernetes.io/projected/4e9ac7ff-c079-4a31-9a4c-7d4f23095492-kube-api-access-vjcvz\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:11:27.940986 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.940951 4541 generic.go:356] "Generic (PLEG): container finished" podID="4e9ac7ff-c079-4a31-9a4c-7d4f23095492" containerID="2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef" exitCode=0 Nov 03 14:11:27.941187 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.941002 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" Nov 03 14:11:27.941187 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.941031 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" event={"ID":"4e9ac7ff-c079-4a31-9a4c-7d4f23095492","Type":"ContainerDied","Data":"2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef"} Nov 03 14:11:27.941187 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.941082 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw" event={"ID":"4e9ac7ff-c079-4a31-9a4c-7d4f23095492","Type":"ContainerDied","Data":"e0b85268020d5ea8a1c0e50dd32367a9eb1f147ebce3cfeb7f88703409e04e3b"} Nov 03 14:11:27.941187 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.941103 4541 scope.go:122] "RemoveContainer" containerID="2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef" Nov 03 14:11:27.951136 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.951118 4541 scope.go:122] "RemoveContainer" containerID="2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef" Nov 03 14:11:27.951407 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:11:27.951389 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef\": container with ID starting with 2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef not found: ID does not exist" containerID="2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef" Nov 03 14:11:27.951470 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:27.951415 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef"} err="failed to get container status \"2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef\": rpc error: code = NotFound desc = could not find container \"2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef\": container with ID starting with 2dbd8f680ab590eea5c57848e3d17a20828b7df8baba7b19c71d9874bbf4caef not found: ID does not exist" Nov 03 14:11:28.099223 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:28.099191 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw"] Nov 03 14:11:28.146012 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:28.145973 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-f11801-7f4bc66bc-bd8tw"] Nov 03 14:11:28.869136 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:11:28.869095 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9ac7ff-c079-4a31-9a4c-7d4f23095492" path="/var/lib/kubelet/pods/4e9ac7ff-c079-4a31-9a4c-7d4f23095492/volumes" Nov 03 14:11:46.892146 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:11:46.892105 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:12:16.890523 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:12:16.890486 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:12:37.335858 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.335813 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f"] Nov 03 14:12:37.652999 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.652957 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf"] Nov 03 14:12:37.653271 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.653238 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" podUID="98513768-8784-4b97-8e08-80396a97c28a" containerName="maas-api" containerID="cri-o://a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba" gracePeriod=30 Nov 03 14:12:37.942534 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.942507 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:12:37.980590 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.980557 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98513768-8784-4b97-8e08-80396a97c28a-kube-api-access-phrx7\" (UniqueName: \"kubernetes.io/projected/98513768-8784-4b97-8e08-80396a97c28a-kube-api-access-phrx7\") pod \"98513768-8784-4b97-8e08-80396a97c28a\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " Nov 03 14:12:37.980817 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.980632 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98513768-8784-4b97-8e08-80396a97c28a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98513768-8784-4b97-8e08-80396a97c28a-openshift-service-ca\") pod \"98513768-8784-4b97-8e08-80396a97c28a\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " Nov 03 14:12:37.980817 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.980677 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-metrics-certs\") pod \"98513768-8784-4b97-8e08-80396a97c28a\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " Nov 03 14:12:37.981027 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.981004 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls\") pod \"98513768-8784-4b97-8e08-80396a97c28a\" (UID: \"98513768-8784-4b97-8e08-80396a97c28a\") " Nov 03 14:12:37.981105 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.981039 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98513768-8784-4b97-8e08-80396a97c28a-openshift-service-ca" pod "98513768-8784-4b97-8e08-80396a97c28a" (UID: "98513768-8784-4b97-8e08-80396a97c28a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:37.981463 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.981438 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98513768-8784-4b97-8e08-80396a97c28a-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:37.982890 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.982862 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-metrics-certs" pod "98513768-8784-4b97-8e08-80396a97c28a" (UID: "98513768-8784-4b97-8e08-80396a97c28a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:37.982993 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.982961 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98513768-8784-4b97-8e08-80396a97c28a-kube-api-access-phrx7" pod "98513768-8784-4b97-8e08-80396a97c28a" (UID: "98513768-8784-4b97-8e08-80396a97c28a"). InnerVolumeSpecName "kube-api-access-phrx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:37.983038 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:37.982995 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls" pod "98513768-8784-4b97-8e08-80396a97c28a" (UID: "98513768-8784-4b97-8e08-80396a97c28a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:12:38.086809 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.083822 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:38.086809 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.083863 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98513768-8784-4b97-8e08-80396a97c28a-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:38.086809 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.083879 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phrx7\" (UniqueName: \"kubernetes.io/projected/98513768-8784-4b97-8e08-80396a97c28a-kube-api-access-phrx7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:38.375200 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.375109 4541 generic.go:356] "Generic (PLEG): container finished" podID="98513768-8784-4b97-8e08-80396a97c28a" containerID="a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba" exitCode=0 Nov 03 14:12:38.375200 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.375155 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" Nov 03 14:12:38.375708 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.375196 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" event={"ID":"98513768-8784-4b97-8e08-80396a97c28a","Type":"ContainerDied","Data":"a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba"} Nov 03 14:12:38.375708 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.375249 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf" event={"ID":"98513768-8784-4b97-8e08-80396a97c28a","Type":"ContainerDied","Data":"994e8750d56c053be3d415ef5a90ecbf62039894230c480b6dc73dc7dfa5a3de"} Nov 03 14:12:38.375708 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.375271 4541 scope.go:122] "RemoveContainer" containerID="a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba" Nov 03 14:12:38.387000 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.386978 4541 scope.go:122] "RemoveContainer" containerID="a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba" Nov 03 14:12:38.387413 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:12:38.387388 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba\": container with ID starting with a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba not found: ID does not exist" containerID="a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba" Nov 03 14:12:38.387472 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.387425 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba"} err="failed to get container status \"a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba\": rpc error: code = NotFound desc = could not find container \"a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba\": container with ID starting with a3bbfd0bc65b9db513e0d79936a1eb1cc3675f58a49478a08a2d62e1210520ba not found: ID does not exist" Nov 03 14:12:38.531424 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.531393 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf"] Nov 03 14:12:38.536063 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.536036 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f11801-6876974bc6-9cdvf"] Nov 03 14:12:38.868579 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:38.868544 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98513768-8784-4b97-8e08-80396a97c28a" path="/var/lib/kubelet/pods/98513768-8784-4b97-8e08-80396a97c28a/volumes" Nov 03 14:12:42.336383 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:42.336284 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" podUID="cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d" containerName="payload-processing" containerID="cri-o://bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf" gracePeriod=25 Nov 03 14:12:42.612061 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:42.612033 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:12:42.636497 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:42.636462 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-kube-api-access-th79w\" (UniqueName: \"kubernetes.io/projected/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-kube-api-access-th79w\") pod \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\" (UID: \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\") " Nov 03 14:12:42.636650 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:42.636518 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-plugins-config-volume\") pod \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\" (UID: \"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d\") " Nov 03 14:12:42.636971 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:42.636945 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-plugins-config-volume" pod "cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d" (UID: "cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:42.638646 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:42.638624 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-kube-api-access-th79w" pod "cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d" (UID: "cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d"). InnerVolumeSpecName "kube-api-access-th79w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:42.738468 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:42.738423 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-th79w\" (UniqueName: \"kubernetes.io/projected/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-kube-api-access-th79w\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:42.738468 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:42.738454 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:43.407322 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:43.407285 4541 generic.go:356] "Generic (PLEG): container finished" podID="cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d" containerID="bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf" exitCode=0 Nov 03 14:12:43.407823 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:43.407336 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" Nov 03 14:12:43.407823 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:43.407366 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" event={"ID":"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d","Type":"ContainerDied","Data":"bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf"} Nov 03 14:12:43.407823 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:43.407423 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f" event={"ID":"cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d","Type":"ContainerDied","Data":"acab883e273f2fd0f2622f83551df7fdc54c76769e7f7f7fc6bdf8dcf6ffa0c9"} Nov 03 14:12:43.407823 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:43.407447 4541 scope.go:122] "RemoveContainer" containerID="bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf" Nov 03 14:12:43.417271 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:43.417250 4541 scope.go:122] "RemoveContainer" containerID="bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf" Nov 03 14:12:43.417536 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:12:43.417513 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf\": container with ID starting with bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf not found: ID does not exist" containerID="bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf" Nov 03 14:12:43.417588 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:43.417545 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf"} err="failed to get container status \"bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf\": rpc error: code = NotFound desc = could not find container \"bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf\": container with ID starting with bd647c677712e044f713ec03507a60964b290be41c0e0fa0bcabb30f2091d8bf not found: ID does not exist" Nov 03 14:12:43.571433 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:43.571403 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f"] Nov 03 14:12:43.627816 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:43.627747 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6375e7-f6db68b95-j2s8f"] Nov 03 14:12:44.869745 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:12:44.869711 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d" path="/var/lib/kubelet/pods/cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d/volumes" Nov 03 14:12:46.901864 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:12:46.901812 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:13:16.890386 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:13:16.890343 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:13:39.134019 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:39.133975 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg"] Nov 03 14:13:39.241839 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:39.241802 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk"] Nov 03 14:13:44.135030 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.134984 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" podUID="8c443546-6048-42d4-8966-d6cfe7242c2b" containerName="payload-processing" containerID="cri-o://b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750" gracePeriod=25 Nov 03 14:13:44.242484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.242439 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" podUID="92466625-51ae-43a0-b3a9-3a035ac33f93" containerName="payload-pre-processing" containerID="cri-o://57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb" gracePeriod=25 Nov 03 14:13:44.413065 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.413036 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:13:44.524721 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.513091 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c443546-6048-42d4-8966-d6cfe7242c2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c443546-6048-42d4-8966-d6cfe7242c2b-plugins-config-volume\") pod \"8c443546-6048-42d4-8966-d6cfe7242c2b\" (UID: \"8c443546-6048-42d4-8966-d6cfe7242c2b\") " Nov 03 14:13:44.525239 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.525215 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c443546-6048-42d4-8966-d6cfe7242c2b-kube-api-access-rc5mh\" (UniqueName: \"kubernetes.io/projected/8c443546-6048-42d4-8966-d6cfe7242c2b-kube-api-access-rc5mh\") pod \"8c443546-6048-42d4-8966-d6cfe7242c2b\" (UID: \"8c443546-6048-42d4-8966-d6cfe7242c2b\") " Nov 03 14:13:44.525660 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.513874 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c443546-6048-42d4-8966-d6cfe7242c2b-plugins-config-volume" pod "8c443546-6048-42d4-8966-d6cfe7242c2b" (UID: "8c443546-6048-42d4-8966-d6cfe7242c2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:13:44.527936 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.527912 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c443546-6048-42d4-8966-d6cfe7242c2b-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:44.555933 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.555536 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c443546-6048-42d4-8966-d6cfe7242c2b-kube-api-access-rc5mh" pod "8c443546-6048-42d4-8966-d6cfe7242c2b" (UID: "8c443546-6048-42d4-8966-d6cfe7242c2b"). InnerVolumeSpecName "kube-api-access-rc5mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:13:44.566485 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.566147 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:13:44.643795 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.643731 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92466625-51ae-43a0-b3a9-3a035ac33f93-kube-api-access-ppb7x\" (UniqueName: \"kubernetes.io/projected/92466625-51ae-43a0-b3a9-3a035ac33f93-kube-api-access-ppb7x\") pod \"92466625-51ae-43a0-b3a9-3a035ac33f93\" (UID: \"92466625-51ae-43a0-b3a9-3a035ac33f93\") " Nov 03 14:13:44.643995 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.643926 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92466625-51ae-43a0-b3a9-3a035ac33f93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92466625-51ae-43a0-b3a9-3a035ac33f93-plugins-config-volume\") pod \"92466625-51ae-43a0-b3a9-3a035ac33f93\" (UID: \"92466625-51ae-43a0-b3a9-3a035ac33f93\") " Nov 03 14:13:44.644798 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.644468 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc5mh\" (UniqueName: \"kubernetes.io/projected/8c443546-6048-42d4-8966-d6cfe7242c2b-kube-api-access-rc5mh\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:44.645807 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.644983 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92466625-51ae-43a0-b3a9-3a035ac33f93-plugins-config-volume" pod "92466625-51ae-43a0-b3a9-3a035ac33f93" (UID: "92466625-51ae-43a0-b3a9-3a035ac33f93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:13:44.657658 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.654923 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92466625-51ae-43a0-b3a9-3a035ac33f93-kube-api-access-ppb7x" pod "92466625-51ae-43a0-b3a9-3a035ac33f93" (UID: "92466625-51ae-43a0-b3a9-3a035ac33f93"). InnerVolumeSpecName "kube-api-access-ppb7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:13:44.748467 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.748331 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppb7x\" (UniqueName: \"kubernetes.io/projected/92466625-51ae-43a0-b3a9-3a035ac33f93-kube-api-access-ppb7x\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:44.748467 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.748369 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92466625-51ae-43a0-b3a9-3a035ac33f93-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:13:44.859942 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.859903 4541 generic.go:356] "Generic (PLEG): container finished" podID="8c443546-6048-42d4-8966-d6cfe7242c2b" containerID="b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750" exitCode=0 Nov 03 14:13:44.860343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.860305 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" event={"ID":"8c443546-6048-42d4-8966-d6cfe7242c2b","Type":"ContainerDied","Data":"b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750"} Nov 03 14:13:44.860584 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.860565 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" event={"ID":"8c443546-6048-42d4-8966-d6cfe7242c2b","Type":"ContainerDied","Data":"d8bf8944db3e31dbd9a19ab1ab549a5d11b6af786fe8cad60dbd0b35054191c4"} Nov 03 14:13:44.860794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.860713 4541 scope.go:122] "RemoveContainer" containerID="b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750" Nov 03 14:13:44.861042 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.861024 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg" Nov 03 14:13:44.872286 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.872240 4541 generic.go:356] "Generic (PLEG): container finished" podID="92466625-51ae-43a0-b3a9-3a035ac33f93" containerID="57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb" exitCode=0 Nov 03 14:13:44.872681 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.872632 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" Nov 03 14:13:44.902531 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.902493 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" event={"ID":"92466625-51ae-43a0-b3a9-3a035ac33f93","Type":"ContainerDied","Data":"57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb"} Nov 03 14:13:44.902798 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.902761 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk" event={"ID":"92466625-51ae-43a0-b3a9-3a035ac33f93","Type":"ContainerDied","Data":"e7d0925d14cbd2f613dd81f795488db70cc72f2befb5db6862099b923e0a46d5"} Nov 03 14:13:44.909548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.909528 4541 scope.go:122] "RemoveContainer" containerID="b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750" Nov 03 14:13:44.909871 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:13:44.909849 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750\": container with ID starting with b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750 not found: ID does not exist" containerID="b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750" Nov 03 14:13:44.909948 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.909878 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750"} err="failed to get container status \"b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750\": rpc error: code = NotFound desc = could not find container \"b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750\": container with ID starting with b6a8ca7efc83d4200a431854ffda9945896390fe4d8f294125e0412e65d1e750 not found: ID does not exist" Nov 03 14:13:44.909948 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.909895 4541 scope.go:122] "RemoveContainer" containerID="57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb" Nov 03 14:13:44.919576 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.919560 4541 scope.go:122] "RemoveContainer" containerID="57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb" Nov 03 14:13:44.919900 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:13:44.919863 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb\": container with ID starting with 57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb not found: ID does not exist" containerID="57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb" Nov 03 14:13:44.919951 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:44.919906 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb"} err="failed to get container status \"57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb\": rpc error: code = NotFound desc = could not find container \"57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb\": container with ID starting with 57210390e23fa87ec8a82649e5c487d0bd0d88118f4229abef55517667b70edb not found: ID does not exist" Nov 03 14:13:45.024347 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:45.024249 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg"] Nov 03 14:13:45.027165 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:45.027133 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b6c8c7-ccd54fcf5-dtkmg"] Nov 03 14:13:45.130916 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:45.130877 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk"] Nov 03 14:13:45.156797 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:45.153359 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b6c8c7-68c8c5964c-2z9vk"] Nov 03 14:13:45.970796 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:45.969845 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:13:45.970796 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:45.969927 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:13:46.874004 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:46.872976 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c443546-6048-42d4-8966-d6cfe7242c2b" path="/var/lib/kubelet/pods/8c443546-6048-42d4-8966-d6cfe7242c2b/volumes" Nov 03 14:13:46.874004 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:13:46.873561 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92466625-51ae-43a0-b3a9-3a035ac33f93" path="/var/lib/kubelet/pods/92466625-51ae-43a0-b3a9-3a035ac33f93/volumes" Nov 03 14:13:46.893000 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:13:46.892957 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:14:16.902378 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:14:16.902285 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:14:24.791844 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.791804 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj"] Nov 03 14:14:24.792374 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792360 4541 cpu_manager.go:414] "containerMap: removing container" podUID="4e9ac7ff-c079-4a31-9a4c-7d4f23095492" containerName="payload-pre-processing" Nov 03 14:14:24.792423 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792375 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9ac7ff-c079-4a31-9a4c-7d4f23095492" containerName="payload-pre-processing" Nov 03 14:14:24.792423 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792390 4541 cpu_manager.go:414] "containerMap: removing container" podUID="98513768-8784-4b97-8e08-80396a97c28a" containerName="maas-api" Nov 03 14:14:24.792423 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792395 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98513768-8784-4b97-8e08-80396a97c28a" containerName="maas-api" Nov 03 14:14:24.792523 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792424 4541 cpu_manager.go:414] "containerMap: removing container" podUID="92466625-51ae-43a0-b3a9-3a035ac33f93" containerName="payload-pre-processing" Nov 03 14:14:24.792523 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792429 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92466625-51ae-43a0-b3a9-3a035ac33f93" containerName="payload-pre-processing" Nov 03 14:14:24.792523 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792444 4541 cpu_manager.go:414] "containerMap: removing container" podUID="cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d" containerName="payload-processing" Nov 03 14:14:24.792523 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792449 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d" containerName="payload-processing" Nov 03 14:14:24.792523 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792465 4541 cpu_manager.go:414] "containerMap: removing container" podUID="8c443546-6048-42d4-8966-d6cfe7242c2b" containerName="payload-processing" Nov 03 14:14:24.792523 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792471 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c443546-6048-42d4-8966-d6cfe7242c2b" containerName="payload-processing" Nov 03 14:14:24.792922 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792547 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="92466625-51ae-43a0-b3a9-3a035ac33f93" containerName="payload-pre-processing" Nov 03 14:14:24.792922 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792557 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="98513768-8784-4b97-8e08-80396a97c28a" containerName="maas-api" Nov 03 14:14:24.792922 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792564 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc7c5a6-f412-41c3-b0d8-8e75ff424e1d" containerName="payload-processing" Nov 03 14:14:24.792922 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792573 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c443546-6048-42d4-8966-d6cfe7242c2b" containerName="payload-processing" Nov 03 14:14:24.792922 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.792580 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9ac7ff-c079-4a31-9a4c-7d4f23095492" containerName="payload-pre-processing" Nov 03 14:14:24.796159 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.796142 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:24.845849 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.845766 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-89273860\"" Nov 03 14:14:24.896827 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.896166 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-89273860-dockercfg-4qlzn\"" Nov 03 14:14:24.901805 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.901749 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj"] Nov 03 14:14:24.976118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.976082 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smqj5\" (UniqueName: \"kubernetes.io/projected/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-kube-api-access-smqj5\") pod \"payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj\" (UID: \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:24.976336 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:24.976306 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj\" (UID: \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:25.077167 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:25.077076 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smqj5\" (UniqueName: \"kubernetes.io/projected/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-kube-api-access-smqj5\") pod \"payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj\" (UID: \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:25.077332 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:25.077275 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj\" (UID: \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:25.078122 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:25.078097 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj\" (UID: \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:25.132128 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:25.132099 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smqj5\" (UniqueName: \"kubernetes.io/projected/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-kube-api-access-smqj5\") pod \"payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj\" (UID: \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:25.409554 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:25.409517 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:25.787428 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:25.787399 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj"] Nov 03 14:14:25.788576 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:14:25.788546 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697c9c9c_206e_44c2_8ded_4e10d15a8ad2.slice/crio-d13859b054cc6758b83301ecc033179552bf292b1bc7a5bebecd374b112d2d4c WatchSource:0}: Error finding container d13859b054cc6758b83301ecc033179552bf292b1bc7a5bebecd374b112d2d4c: Status 404 returned error can't find the container with id d13859b054cc6758b83301ecc033179552bf292b1bc7a5bebecd374b112d2d4c Nov 03 14:14:26.174343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:26.174304 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" event={"ID":"697c9c9c-206e-44c2-8ded-4e10d15a8ad2","Type":"ContainerStarted","Data":"7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0"} Nov 03 14:14:26.174885 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:26.174386 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:26.174885 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:26.174403 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" event={"ID":"697c9c9c-206e-44c2-8ded-4e10d15a8ad2","Type":"ContainerStarted","Data":"d13859b054cc6758b83301ecc033179552bf292b1bc7a5bebecd374b112d2d4c"} Nov 03 14:14:26.299861 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:26.299797 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" podStartSLOduration=2.299757326 podStartE2EDuration="2.299757326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:26.298038645 +0000 UTC m=+4197.612010505" watchObservedRunningTime="2026-11-03 14:14:26.299757326 +0000 UTC m=+4197.613729186" Nov 03 14:14:29.401267 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:29.401232 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:14:29.409023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:29.408999 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:14:37.183111 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:14:37.183077 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:14:46.890492 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:14:46.890447 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:15:00.232134 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.232089 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp"] Nov 03 14:15:00.237857 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.237828 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:00.261705 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.261673 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b"] Nov 03 14:15:00.271012 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.270971 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t"] Nov 03 14:15:00.271392 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.271369 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:00.275734 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.275628 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:00.280337 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.280030 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmgs2\" (UniqueName: \"kubernetes.io/projected/60097229-aca6-41fc-82e7-9f523a144f87-kube-api-access-hmgs2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp\" (UID: \"60097229-aca6-41fc-82e7-9f523a144f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:00.280337 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.280081 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60097229-aca6-41fc-82e7-9f523a144f87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp\" (UID: \"60097229-aca6-41fc-82e7-9f523a144f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:00.353461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.349567 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp"] Nov 03 14:15:00.366795 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.366719 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:15:00.384678 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.384627 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04cad50-17c0-4059-956f-54212b1f0d08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b\" (UID: \"d04cad50-17c0-4059-956f-54212b1f0d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:00.384876 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.384695 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmgs2\" (UniqueName: \"kubernetes.io/projected/60097229-aca6-41fc-82e7-9f523a144f87-kube-api-access-hmgs2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp\" (UID: \"60097229-aca6-41fc-82e7-9f523a144f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:00.384876 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.384853 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60097229-aca6-41fc-82e7-9f523a144f87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp\" (UID: \"60097229-aca6-41fc-82e7-9f523a144f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:00.385437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.385213 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrpk\" (UniqueName: \"kubernetes.io/projected/d04cad50-17c0-4059-956f-54212b1f0d08-kube-api-access-mlrpk\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b\" (UID: \"d04cad50-17c0-4059-956f-54212b1f0d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:00.385437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.385320 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dzc9\" (UniqueName: \"kubernetes.io/projected/7f151f83-66bc-4d9f-ab3d-29fb143ad893-kube-api-access-4dzc9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t\" (UID: \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:00.385437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.385416 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f151f83-66bc-4d9f-ab3d-29fb143ad893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t\" (UID: \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:00.400031 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.400003 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60097229-aca6-41fc-82e7-9f523a144f87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp\" (UID: \"60097229-aca6-41fc-82e7-9f523a144f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:00.412137 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.405809 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b"] Nov 03 14:15:00.422983 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.422959 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t"] Nov 03 14:15:00.467862 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.467831 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmgs2\" (UniqueName: \"kubernetes.io/projected/60097229-aca6-41fc-82e7-9f523a144f87-kube-api-access-hmgs2\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp\" (UID: \"60097229-aca6-41fc-82e7-9f523a144f87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:00.490835 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.486800 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f151f83-66bc-4d9f-ab3d-29fb143ad893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t\" (UID: \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:00.490835 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.486884 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04cad50-17c0-4059-956f-54212b1f0d08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b\" (UID: \"d04cad50-17c0-4059-956f-54212b1f0d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:00.490835 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.486996 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrpk\" (UniqueName: \"kubernetes.io/projected/d04cad50-17c0-4059-956f-54212b1f0d08-kube-api-access-mlrpk\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b\" (UID: \"d04cad50-17c0-4059-956f-54212b1f0d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:00.490835 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.487078 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzc9\" (UniqueName: \"kubernetes.io/projected/7f151f83-66bc-4d9f-ab3d-29fb143ad893-kube-api-access-4dzc9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t\" (UID: \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:00.490835 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.488210 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f151f83-66bc-4d9f-ab3d-29fb143ad893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t\" (UID: \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:00.490835 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.488915 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04cad50-17c0-4059-956f-54212b1f0d08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b\" (UID: \"d04cad50-17c0-4059-956f-54212b1f0d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:00.570797 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.569058 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:00.594754 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.581446 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrpk\" (UniqueName: \"kubernetes.io/projected/d04cad50-17c0-4059-956f-54212b1f0d08-kube-api-access-mlrpk\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b\" (UID: \"d04cad50-17c0-4059-956f-54212b1f0d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:00.616799 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.616349 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:00.626941 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.623123 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dzc9\" (UniqueName: \"kubernetes.io/projected/7f151f83-66bc-4d9f-ab3d-29fb143ad893-kube-api-access-4dzc9\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t\" (UID: \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:00.657793 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:00.654969 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:01.430476 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:01.430438 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp"] Nov 03 14:15:01.432425 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:15:01.432389 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60097229_aca6_41fc_82e7_9f523a144f87.slice/crio-6cbce9800185b83e03bf027c511a2ac6c195fdd8c86de502db56d45738e4a280 WatchSource:0}: Error finding container 6cbce9800185b83e03bf027c511a2ac6c195fdd8c86de502db56d45738e4a280: Status 404 returned error can't find the container with id 6cbce9800185b83e03bf027c511a2ac6c195fdd8c86de502db56d45738e4a280 Nov 03 14:15:01.720850 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:01.713113 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b"] Nov 03 14:15:01.786107 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:01.784551 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t"] Nov 03 14:15:02.451371 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:02.451335 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" event={"ID":"7f151f83-66bc-4d9f-ab3d-29fb143ad893","Type":"ContainerStarted","Data":"2b66e6a6d33d5fad8c8f035c514ca686ce0ebd418ebc3dd7abe6d4e6eb521eae"} Nov 03 14:15:02.453051 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:02.453020 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" event={"ID":"60097229-aca6-41fc-82e7-9f523a144f87","Type":"ContainerStarted","Data":"414b9e91af590d04055b1314b29464a4bd172eca4630e6fc750ec09513169c13"} Nov 03 14:15:02.453150 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:02.453081 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" event={"ID":"60097229-aca6-41fc-82e7-9f523a144f87","Type":"ContainerStarted","Data":"6cbce9800185b83e03bf027c511a2ac6c195fdd8c86de502db56d45738e4a280"} Nov 03 14:15:02.454785 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:02.454748 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" event={"ID":"d04cad50-17c0-4059-956f-54212b1f0d08","Type":"ContainerStarted","Data":"7f1e537539fd2af11efe9a8fdc35d488a8529412bf61bd95df2070e4a215da95"} Nov 03 14:15:02.454895 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:02.454807 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" event={"ID":"d04cad50-17c0-4059-956f-54212b1f0d08","Type":"ContainerStarted","Data":"45ce24a8140b4c5f30861a339e9fd781b5f9c70c856a86ff69877343d617f935"} Nov 03 14:15:02.568082 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:02.567991 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" podStartSLOduration=2.567978482 podStartE2EDuration="2.567978482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:02.564542249 +0000 UTC m=+4233.878514336" watchObservedRunningTime="2026-11-03 14:15:02.567978482 +0000 UTC m=+4233.881950341" Nov 03 14:15:02.676693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:02.676553 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" podStartSLOduration=2.676533394 podStartE2EDuration="2.676533394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:02.675135759 +0000 UTC m=+4233.989107621" watchObservedRunningTime="2026-11-03 14:15:02.676533394 +0000 UTC m=+4233.990505255" Nov 03 14:15:03.463786 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:03.463743 4541 generic.go:356] "Generic (PLEG): container finished" podID="d04cad50-17c0-4059-956f-54212b1f0d08" containerID="7f1e537539fd2af11efe9a8fdc35d488a8529412bf61bd95df2070e4a215da95" exitCode=60 Nov 03 14:15:03.464226 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:03.463835 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" event={"ID":"d04cad50-17c0-4059-956f-54212b1f0d08","Type":"ContainerDied","Data":"7f1e537539fd2af11efe9a8fdc35d488a8529412bf61bd95df2070e4a215da95"} Nov 03 14:15:03.464226 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:03.464129 4541 scope.go:122] "RemoveContainer" containerID="7f1e537539fd2af11efe9a8fdc35d488a8529412bf61bd95df2070e4a215da95" Nov 03 14:15:03.466977 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:03.466949 4541 generic.go:356] "Generic (PLEG): container finished" podID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerID="1a21e9fd9e882f69564d74200367ada33d968729b5867bd236d48a55043bff8b" exitCode=60 Nov 03 14:15:03.467070 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:03.467030 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" event={"ID":"7f151f83-66bc-4d9f-ab3d-29fb143ad893","Type":"ContainerDied","Data":"1a21e9fd9e882f69564d74200367ada33d968729b5867bd236d48a55043bff8b"} Nov 03 14:15:03.467260 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:03.467242 4541 scope.go:122] "RemoveContainer" containerID="1a21e9fd9e882f69564d74200367ada33d968729b5867bd236d48a55043bff8b" Nov 03 14:15:03.469020 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:03.468997 4541 generic.go:356] "Generic (PLEG): container finished" podID="60097229-aca6-41fc-82e7-9f523a144f87" containerID="414b9e91af590d04055b1314b29464a4bd172eca4630e6fc750ec09513169c13" exitCode=60 Nov 03 14:15:03.469107 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:03.469032 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" event={"ID":"60097229-aca6-41fc-82e7-9f523a144f87","Type":"ContainerDied","Data":"414b9e91af590d04055b1314b29464a4bd172eca4630e6fc750ec09513169c13"} Nov 03 14:15:03.469301 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:03.469284 4541 scope.go:122] "RemoveContainer" containerID="414b9e91af590d04055b1314b29464a4bd172eca4630e6fc750ec09513169c13" Nov 03 14:15:04.487972 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:04.487923 4541 generic.go:356] "Generic (PLEG): container finished" podID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerID="9e678e43e3695bda63383612c114dd074449db41624e28001b04d75ec38c939a" exitCode=60 Nov 03 14:15:04.488436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:04.488015 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" event={"ID":"7f151f83-66bc-4d9f-ab3d-29fb143ad893","Type":"ContainerDied","Data":"9e678e43e3695bda63383612c114dd074449db41624e28001b04d75ec38c939a"} Nov 03 14:15:04.488436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:04.488090 4541 scope.go:122] "RemoveContainer" containerID="1a21e9fd9e882f69564d74200367ada33d968729b5867bd236d48a55043bff8b" Nov 03 14:15:04.488436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:04.488390 4541 scope.go:122] "RemoveContainer" containerID="9e678e43e3695bda63383612c114dd074449db41624e28001b04d75ec38c939a" Nov 03 14:15:04.488862 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:04.488832 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t_odh-ai-gateway-infra(7f151f83-66bc-4d9f-ab3d-29fb143ad893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" Nov 03 14:15:04.491825 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:04.491798 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" event={"ID":"60097229-aca6-41fc-82e7-9f523a144f87","Type":"ContainerStarted","Data":"2a35ed7dd7eb9263997f4d751a268a1290b3b2cb3cb7ee38eb8dc720b1af89d0"} Nov 03 14:15:04.493847 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:04.493815 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" event={"ID":"d04cad50-17c0-4059-956f-54212b1f0d08","Type":"ContainerStarted","Data":"5b867cc2b09d6141063a3abe7e5d360e76b7957abffa2962cce77f17232f76c5"} Nov 03 14:15:05.521804 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:05.521555 4541 scope.go:122] "RemoveContainer" containerID="9e678e43e3695bda63383612c114dd074449db41624e28001b04d75ec38c939a" Nov 03 14:15:05.522320 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:05.521938 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t_odh-ai-gateway-infra(7f151f83-66bc-4d9f-ab3d-29fb143ad893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" Nov 03 14:15:05.540789 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:05.539949 4541 generic.go:356] "Generic (PLEG): container finished" podID="60097229-aca6-41fc-82e7-9f523a144f87" containerID="2a35ed7dd7eb9263997f4d751a268a1290b3b2cb3cb7ee38eb8dc720b1af89d0" exitCode=60 Nov 03 14:15:05.540789 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:05.540043 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" event={"ID":"60097229-aca6-41fc-82e7-9f523a144f87","Type":"ContainerDied","Data":"2a35ed7dd7eb9263997f4d751a268a1290b3b2cb3cb7ee38eb8dc720b1af89d0"} Nov 03 14:15:05.540789 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:05.540098 4541 scope.go:122] "RemoveContainer" containerID="414b9e91af590d04055b1314b29464a4bd172eca4630e6fc750ec09513169c13" Nov 03 14:15:05.540789 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:05.540524 4541 scope.go:122] "RemoveContainer" containerID="2a35ed7dd7eb9263997f4d751a268a1290b3b2cb3cb7ee38eb8dc720b1af89d0" Nov 03 14:15:05.541125 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:05.540871 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp_odh-ai-gateway-infra(60097229-aca6-41fc-82e7-9f523a144f87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" podUID="60097229-aca6-41fc-82e7-9f523a144f87" Nov 03 14:15:05.664896 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:05.664846 4541 generic.go:356] "Generic (PLEG): container finished" podID="d04cad50-17c0-4059-956f-54212b1f0d08" containerID="5b867cc2b09d6141063a3abe7e5d360e76b7957abffa2962cce77f17232f76c5" exitCode=60 Nov 03 14:15:05.665632 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:05.665589 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" event={"ID":"d04cad50-17c0-4059-956f-54212b1f0d08","Type":"ContainerDied","Data":"5b867cc2b09d6141063a3abe7e5d360e76b7957abffa2962cce77f17232f76c5"} Nov 03 14:15:05.665928 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:05.665909 4541 scope.go:122] "RemoveContainer" containerID="7f1e537539fd2af11efe9a8fdc35d488a8529412bf61bd95df2070e4a215da95" Nov 03 14:15:05.671218 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:05.671198 4541 scope.go:122] "RemoveContainer" containerID="5b867cc2b09d6141063a3abe7e5d360e76b7957abffa2962cce77f17232f76c5" Nov 03 14:15:05.671690 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:05.671666 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b_odh-ai-gateway-infra(d04cad50-17c0-4059-956f-54212b1f0d08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" Nov 03 14:15:06.672979 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:06.672952 4541 scope.go:122] "RemoveContainer" containerID="2a35ed7dd7eb9263997f4d751a268a1290b3b2cb3cb7ee38eb8dc720b1af89d0" Nov 03 14:15:06.673509 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:06.673296 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp_odh-ai-gateway-infra(60097229-aca6-41fc-82e7-9f523a144f87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" podUID="60097229-aca6-41fc-82e7-9f523a144f87" Nov 03 14:15:06.674461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:06.674438 4541 scope.go:122] "RemoveContainer" containerID="5b867cc2b09d6141063a3abe7e5d360e76b7957abffa2962cce77f17232f76c5" Nov 03 14:15:06.674713 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:06.674692 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b_odh-ai-gateway-infra(d04cad50-17c0-4059-956f-54212b1f0d08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" Nov 03 14:15:15.861649 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:15.861604 4541 scope.go:122] "RemoveContainer" containerID="2a35ed7dd7eb9263997f4d751a268a1290b3b2cb3cb7ee38eb8dc720b1af89d0" Nov 03 14:15:15.862089 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:15.861677 4541 scope.go:122] "RemoveContainer" containerID="9e678e43e3695bda63383612c114dd074449db41624e28001b04d75ec38c939a" Nov 03 14:15:15.862089 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:15.861756 4541 scope.go:122] "RemoveContainer" containerID="5b867cc2b09d6141063a3abe7e5d360e76b7957abffa2962cce77f17232f76c5" Nov 03 14:15:16.736560 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:16.736514 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" event={"ID":"7f151f83-66bc-4d9f-ab3d-29fb143ad893","Type":"ContainerStarted","Data":"679cc6e8c3516f8fa08e9ca1a8785bb06f2d25695c8fe8a30d468dcb3fc44f71"} Nov 03 14:15:16.739081 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:16.739050 4541 generic.go:356] "Generic (PLEG): container finished" podID="60097229-aca6-41fc-82e7-9f523a144f87" containerID="5662b990aa69c574b064147808fab6b3b4f75d5fe335d1873cf41bb81cf8e5ae" exitCode=60 Nov 03 14:15:16.739436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:16.739100 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" event={"ID":"60097229-aca6-41fc-82e7-9f523a144f87","Type":"ContainerDied","Data":"5662b990aa69c574b064147808fab6b3b4f75d5fe335d1873cf41bb81cf8e5ae"} Nov 03 14:15:16.739436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:16.739166 4541 scope.go:122] "RemoveContainer" containerID="2a35ed7dd7eb9263997f4d751a268a1290b3b2cb3cb7ee38eb8dc720b1af89d0" Nov 03 14:15:16.739814 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:16.739677 4541 scope.go:122] "RemoveContainer" containerID="5662b990aa69c574b064147808fab6b3b4f75d5fe335d1873cf41bb81cf8e5ae" Nov 03 14:15:16.740023 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:16.739998 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp_odh-ai-gateway-infra(60097229-aca6-41fc-82e7-9f523a144f87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" podUID="60097229-aca6-41fc-82e7-9f523a144f87" Nov 03 14:15:16.741891 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:16.741867 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" event={"ID":"d04cad50-17c0-4059-956f-54212b1f0d08","Type":"ContainerStarted","Data":"1bf3ddab92bc406d49243b27e91066ea742446e9f1b3ad56617835d71674233e"} Nov 03 14:15:16.847694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:16.847646 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" podStartSLOduration=16.8476319 podStartE2EDuration="16.8476319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:16.844693674 +0000 UTC m=+4248.158665528" watchObservedRunningTime="2026-11-03 14:15:16.8476319 +0000 UTC m=+4248.161603773" Nov 03 14:15:16.889750 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:16.889713 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:15:17.749684 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.749651 4541 generic.go:356] "Generic (PLEG): container finished" podID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerID="679cc6e8c3516f8fa08e9ca1a8785bb06f2d25695c8fe8a30d468dcb3fc44f71" exitCode=60 Nov 03 14:15:17.749975 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.749726 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" event={"ID":"7f151f83-66bc-4d9f-ab3d-29fb143ad893","Type":"ContainerDied","Data":"679cc6e8c3516f8fa08e9ca1a8785bb06f2d25695c8fe8a30d468dcb3fc44f71"} Nov 03 14:15:17.749975 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.749810 4541 scope.go:122] "RemoveContainer" containerID="9e678e43e3695bda63383612c114dd074449db41624e28001b04d75ec38c939a" Nov 03 14:15:17.750208 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.750186 4541 scope.go:122] "RemoveContainer" containerID="679cc6e8c3516f8fa08e9ca1a8785bb06f2d25695c8fe8a30d468dcb3fc44f71" Nov 03 14:15:17.750511 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:17.750487 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t_odh-ai-gateway-infra(7f151f83-66bc-4d9f-ab3d-29fb143ad893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" Nov 03 14:15:17.753051 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.753031 4541 generic.go:356] "Generic (PLEG): container finished" podID="d04cad50-17c0-4059-956f-54212b1f0d08" containerID="1bf3ddab92bc406d49243b27e91066ea742446e9f1b3ad56617835d71674233e" exitCode=60 Nov 03 14:15:17.753158 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.753109 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" event={"ID":"d04cad50-17c0-4059-956f-54212b1f0d08","Type":"ContainerDied","Data":"1bf3ddab92bc406d49243b27e91066ea742446e9f1b3ad56617835d71674233e"} Nov 03 14:15:17.753458 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.753441 4541 scope.go:122] "RemoveContainer" containerID="1bf3ddab92bc406d49243b27e91066ea742446e9f1b3ad56617835d71674233e" Nov 03 14:15:17.753706 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:17.753686 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b_odh-ai-gateway-infra(d04cad50-17c0-4059-956f-54212b1f0d08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" Nov 03 14:15:17.761675 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.761657 4541 scope.go:122] "RemoveContainer" containerID="5b867cc2b09d6141063a3abe7e5d360e76b7957abffa2962cce77f17232f76c5" Nov 03 14:15:17.855851 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.855816 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t"] Nov 03 14:15:17.970814 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:17.970780 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp"] Nov 03 14:15:18.083758 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.083721 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b"] Nov 03 14:15:18.143389 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.143359 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:18.242919 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.242884 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60097229-aca6-41fc-82e7-9f523a144f87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60097229-aca6-41fc-82e7-9f523a144f87-maas-api-service-ca\") pod \"60097229-aca6-41fc-82e7-9f523a144f87\" (UID: \"60097229-aca6-41fc-82e7-9f523a144f87\") " Nov 03 14:15:18.243098 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.243084 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60097229-aca6-41fc-82e7-9f523a144f87-kube-api-access-hmgs2\" (UniqueName: \"kubernetes.io/projected/60097229-aca6-41fc-82e7-9f523a144f87-kube-api-access-hmgs2\") pod \"60097229-aca6-41fc-82e7-9f523a144f87\" (UID: \"60097229-aca6-41fc-82e7-9f523a144f87\") " Nov 03 14:15:18.243322 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.243294 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60097229-aca6-41fc-82e7-9f523a144f87-maas-api-service-ca" pod "60097229-aca6-41fc-82e7-9f523a144f87" (UID: "60097229-aca6-41fc-82e7-9f523a144f87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:18.243525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.243503 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60097229-aca6-41fc-82e7-9f523a144f87-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:18.245192 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.245169 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60097229-aca6-41fc-82e7-9f523a144f87-kube-api-access-hmgs2" pod "60097229-aca6-41fc-82e7-9f523a144f87" (UID: "60097229-aca6-41fc-82e7-9f523a144f87"). InnerVolumeSpecName "kube-api-access-hmgs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:18.344542 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.344460 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmgs2\" (UniqueName: \"kubernetes.io/projected/60097229-aca6-41fc-82e7-9f523a144f87-kube-api-access-hmgs2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:18.761671 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.761636 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" event={"ID":"60097229-aca6-41fc-82e7-9f523a144f87","Type":"ContainerDied","Data":"6cbce9800185b83e03bf027c511a2ac6c195fdd8c86de502db56d45738e4a280"} Nov 03 14:15:18.761824 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.761643 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp" Nov 03 14:15:18.761824 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.761697 4541 scope.go:122] "RemoveContainer" containerID="5662b990aa69c574b064147808fab6b3b4f75d5fe335d1873cf41bb81cf8e5ae" Nov 03 14:15:18.932649 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.932623 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:18.935597 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.935572 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:18.976828 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.976795 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp"] Nov 03 14:15:18.987069 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:18.987037 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895255-7q5kp"] Nov 03 14:15:19.053510 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.053411 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f151f83-66bc-4d9f-ab3d-29fb143ad893-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f151f83-66bc-4d9f-ab3d-29fb143ad893-maas-api-service-ca\") pod \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\" (UID: \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\") " Nov 03 14:15:19.053510 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.053494 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d04cad50-17c0-4059-956f-54212b1f0d08-kube-api-access-mlrpk\" (UniqueName: \"kubernetes.io/projected/d04cad50-17c0-4059-956f-54212b1f0d08-kube-api-access-mlrpk\") pod \"d04cad50-17c0-4059-956f-54212b1f0d08\" (UID: \"d04cad50-17c0-4059-956f-54212b1f0d08\") " Nov 03 14:15:19.053785 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.053561 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f151f83-66bc-4d9f-ab3d-29fb143ad893-kube-api-access-4dzc9\" (UniqueName: \"kubernetes.io/projected/7f151f83-66bc-4d9f-ab3d-29fb143ad893-kube-api-access-4dzc9\") pod \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\" (UID: \"7f151f83-66bc-4d9f-ab3d-29fb143ad893\") " Nov 03 14:15:19.053785 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.053643 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d04cad50-17c0-4059-956f-54212b1f0d08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04cad50-17c0-4059-956f-54212b1f0d08-maas-api-service-ca\") pod \"d04cad50-17c0-4059-956f-54212b1f0d08\" (UID: \"d04cad50-17c0-4059-956f-54212b1f0d08\") " Nov 03 14:15:19.053923 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.053895 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f151f83-66bc-4d9f-ab3d-29fb143ad893-maas-api-service-ca" pod "7f151f83-66bc-4d9f-ab3d-29fb143ad893" (UID: "7f151f83-66bc-4d9f-ab3d-29fb143ad893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:19.054011 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.053992 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d04cad50-17c0-4059-956f-54212b1f0d08-maas-api-service-ca" pod "d04cad50-17c0-4059-956f-54212b1f0d08" (UID: "d04cad50-17c0-4059-956f-54212b1f0d08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:19.054105 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.054090 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f151f83-66bc-4d9f-ab3d-29fb143ad893-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:19.055856 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.055826 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f151f83-66bc-4d9f-ab3d-29fb143ad893-kube-api-access-4dzc9" pod "7f151f83-66bc-4d9f-ab3d-29fb143ad893" (UID: "7f151f83-66bc-4d9f-ab3d-29fb143ad893"). InnerVolumeSpecName "kube-api-access-4dzc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:19.055967 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.055860 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04cad50-17c0-4059-956f-54212b1f0d08-kube-api-access-mlrpk" pod "d04cad50-17c0-4059-956f-54212b1f0d08" (UID: "d04cad50-17c0-4059-956f-54212b1f0d08"). InnerVolumeSpecName "kube-api-access-mlrpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:19.155849 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.155810 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d04cad50-17c0-4059-956f-54212b1f0d08-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:19.155849 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.155840 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlrpk\" (UniqueName: \"kubernetes.io/projected/d04cad50-17c0-4059-956f-54212b1f0d08-kube-api-access-mlrpk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:19.155849 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.155853 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dzc9\" (UniqueName: \"kubernetes.io/projected/7f151f83-66bc-4d9f-ab3d-29fb143ad893-kube-api-access-4dzc9\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:19.771389 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.771355 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" Nov 03 14:15:19.771621 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.771368 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t" event={"ID":"7f151f83-66bc-4d9f-ab3d-29fb143ad893","Type":"ContainerDied","Data":"2b66e6a6d33d5fad8c8f035c514ca686ce0ebd418ebc3dd7abe6d4e6eb521eae"} Nov 03 14:15:19.771811 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.771759 4541 scope.go:122] "RemoveContainer" containerID="679cc6e8c3516f8fa08e9ca1a8785bb06f2d25695c8fe8a30d468dcb3fc44f71" Nov 03 14:15:19.774555 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.774535 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" Nov 03 14:15:19.774757 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.774566 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b" event={"ID":"d04cad50-17c0-4059-956f-54212b1f0d08","Type":"ContainerDied","Data":"45ce24a8140b4c5f30861a339e9fd781b5f9c70c856a86ff69877343d617f935"} Nov 03 14:15:19.782371 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.782349 4541 scope.go:122] "RemoveContainer" containerID="1bf3ddab92bc406d49243b27e91066ea742446e9f1b3ad56617835d71674233e" Nov 03 14:15:19.928895 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.928856 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b"] Nov 03 14:15:19.931630 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:19.931599 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895255-zbc9b"] Nov 03 14:15:20.033464 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:20.033374 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t"] Nov 03 14:15:20.036234 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:20.036210 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-efbdec-29895255-cvv5t"] Nov 03 14:15:20.868859 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:20.868825 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60097229-aca6-41fc-82e7-9f523a144f87" path="/var/lib/kubelet/pods/60097229-aca6-41fc-82e7-9f523a144f87/volumes" Nov 03 14:15:20.869267 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:20.869253 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" path="/var/lib/kubelet/pods/7f151f83-66bc-4d9f-ab3d-29fb143ad893/volumes" Nov 03 14:15:20.869619 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:15:20.869607 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" path="/var/lib/kubelet/pods/d04cad50-17c0-4059-956f-54212b1f0d08/volumes" Nov 03 14:15:46.892519 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:15:46.892479 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:16:16.890437 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:16:16.890400 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:16:46.889615 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:16:46.889578 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:17:10.024798 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.024683 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn"] Nov 03 14:17:10.025586 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025563 4541 cpu_manager.go:414] "containerMap: removing container" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerName="cleanup" Nov 03 14:17:10.025694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025587 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerName="cleanup" Nov 03 14:17:10.025694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025614 4541 cpu_manager.go:414] "containerMap: removing container" podUID="60097229-aca6-41fc-82e7-9f523a144f87" containerName="cleanup" Nov 03 14:17:10.025694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025623 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60097229-aca6-41fc-82e7-9f523a144f87" containerName="cleanup" Nov 03 14:17:10.025694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025639 4541 cpu_manager.go:414] "containerMap: removing container" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" containerName="cleanup" Nov 03 14:17:10.025694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025647 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" containerName="cleanup" Nov 03 14:17:10.025694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025657 4541 cpu_manager.go:414] "containerMap: removing container" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" containerName="cleanup" Nov 03 14:17:10.025694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025664 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" containerName="cleanup" Nov 03 14:17:10.025694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025679 4541 cpu_manager.go:414] "containerMap: removing container" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" containerName="cleanup" Nov 03 14:17:10.025694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025686 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025714 4541 cpu_manager.go:414] "containerMap: removing container" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025722 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025755 4541 cpu_manager.go:414] "containerMap: removing container" podUID="60097229-aca6-41fc-82e7-9f523a144f87" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025764 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60097229-aca6-41fc-82e7-9f523a144f87" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025925 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025941 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="60097229-aca6-41fc-82e7-9f523a144f87" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025953 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="60097229-aca6-41fc-82e7-9f523a144f87" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025963 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025975 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025983 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" containerName="cleanup" Nov 03 14:17:10.026221 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.025995 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="d04cad50-17c0-4059-956f-54212b1f0d08" containerName="cleanup" Nov 03 14:17:10.029629 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.029608 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.126452 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.126426 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b5daee-serving-cert\"" Nov 03 14:17:10.129460 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.129433 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn"] Nov 03 14:17:10.183014 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.182984 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2798c769-768f-4eff-9944-61f6dc97e873-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.183157 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.183134 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-maas-api-tls\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.183222 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.183172 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-metrics-certs\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.183373 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.183355 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mzx2\" (UniqueName: \"kubernetes.io/projected/2798c769-768f-4eff-9944-61f6dc97e873-kube-api-access-2mzx2\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.285069 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.284986 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-maas-api-tls\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.285069 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.285020 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-metrics-certs\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.285308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.285116 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mzx2\" (UniqueName: \"kubernetes.io/projected/2798c769-768f-4eff-9944-61f6dc97e873-kube-api-access-2mzx2\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.285308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.285209 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2798c769-768f-4eff-9944-61f6dc97e873-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.285955 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.285931 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2798c769-768f-4eff-9944-61f6dc97e873-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.287575 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.287551 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-maas-api-tls\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.287912 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.287893 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-metrics-certs\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.338834 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.338807 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mzx2\" (UniqueName: \"kubernetes.io/projected/2798c769-768f-4eff-9944-61f6dc97e873-kube-api-access-2mzx2\") pod \"maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.342595 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.342578 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:10.715792 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.715743 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn"] Nov 03 14:17:10.716320 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:17:10.716279 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2798c769_768f_4eff_9944_61f6dc97e873.slice/crio-d46664b66935b1932edb3fe8b2e6e22de45e6f8bb7112e0132c9922e64555456 WatchSource:0}: Error finding container d46664b66935b1932edb3fe8b2e6e22de45e6f8bb7112e0132c9922e64555456: Status 404 returned error can't find the container with id d46664b66935b1932edb3fe8b2e6e22de45e6f8bb7112e0132c9922e64555456 Nov 03 14:17:10.717764 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:10.717746 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:17:11.465819 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:11.465788 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" event={"ID":"2798c769-768f-4eff-9944-61f6dc97e873","Type":"ContainerStarted","Data":"d46664b66935b1932edb3fe8b2e6e22de45e6f8bb7112e0132c9922e64555456"} Nov 03 14:17:12.472622 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:12.472590 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" event={"ID":"2798c769-768f-4eff-9944-61f6dc97e873","Type":"ContainerStarted","Data":"2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6"} Nov 03 14:17:12.473109 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:12.472795 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:12.584397 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:12.584348 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" podStartSLOduration=2.494426497 podStartE2EDuration="3.58433588s" totalImagesPullingTime="1.089909383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:17:12.581450617 +0000 UTC m=+4363.895422478" watchObservedRunningTime="2026-11-03 14:17:12.58433588 +0000 UTC m=+4363.898307739" Nov 03 14:17:16.889509 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:17:16.889473 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:17:18.485321 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:18.485291 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:17:28.649329 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:28.649292 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj"] Nov 03 14:17:33.650218 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:33.650170 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" podUID="697c9c9c-206e-44c2-8ded-4e10d15a8ad2" containerName="payload-pre-processing" containerID="cri-o://7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0" gracePeriod=25 Nov 03 14:17:33.922376 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:33.922352 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:17:33.983937 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:33.983907 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-kube-api-access-smqj5\" (UniqueName: \"kubernetes.io/projected/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-kube-api-access-smqj5\") pod \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\" (UID: \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\") " Nov 03 14:17:33.984111 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:33.984012 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-plugins-config-volume\") pod \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\" (UID: \"697c9c9c-206e-44c2-8ded-4e10d15a8ad2\") " Nov 03 14:17:33.984393 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:33.984368 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-plugins-config-volume" pod "697c9c9c-206e-44c2-8ded-4e10d15a8ad2" (UID: "697c9c9c-206e-44c2-8ded-4e10d15a8ad2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:17:33.984691 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:33.984670 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:17:33.986044 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:33.986024 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-kube-api-access-smqj5" pod "697c9c9c-206e-44c2-8ded-4e10d15a8ad2" (UID: "697c9c9c-206e-44c2-8ded-4e10d15a8ad2"). InnerVolumeSpecName "kube-api-access-smqj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:17:34.085900 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.085864 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smqj5\" (UniqueName: \"kubernetes.io/projected/697c9c9c-206e-44c2-8ded-4e10d15a8ad2-kube-api-access-smqj5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:17:34.606754 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.606714 4541 generic.go:356] "Generic (PLEG): container finished" podID="697c9c9c-206e-44c2-8ded-4e10d15a8ad2" containerID="7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0" exitCode=0 Nov 03 14:17:34.607039 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.606801 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" event={"ID":"697c9c9c-206e-44c2-8ded-4e10d15a8ad2","Type":"ContainerDied","Data":"7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0"} Nov 03 14:17:34.607039 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.606825 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" Nov 03 14:17:34.607039 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.606860 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj" event={"ID":"697c9c9c-206e-44c2-8ded-4e10d15a8ad2","Type":"ContainerDied","Data":"d13859b054cc6758b83301ecc033179552bf292b1bc7a5bebecd374b112d2d4c"} Nov 03 14:17:34.607039 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.606886 4541 scope.go:122] "RemoveContainer" containerID="7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0" Nov 03 14:17:34.617090 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.617071 4541 scope.go:122] "RemoveContainer" containerID="7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0" Nov 03 14:17:34.617398 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:17:34.617367 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0\": container with ID starting with 7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0 not found: ID does not exist" containerID="7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0" Nov 03 14:17:34.617495 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.617407 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0"} err="failed to get container status \"7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0\": rpc error: code = NotFound desc = could not find container \"7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0\": container with ID starting with 7f968bb9ae35bd997269c198f0cc8a4f2463b52dbb8782d9119aa2172debb6b0 not found: ID does not exist" Nov 03 14:17:34.768038 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.768003 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj"] Nov 03 14:17:34.814614 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.814579 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-89273860-6d5668f5cf-766xj"] Nov 03 14:17:34.868306 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:17:34.868234 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="697c9c9c-206e-44c2-8ded-4e10d15a8ad2" path="/var/lib/kubelet/pods/697c9c9c-206e-44c2-8ded-4e10d15a8ad2/volumes" Nov 03 14:17:46.890729 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:17:46.890692 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:18:16.890241 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:18:16.890203 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:18:46.895112 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:18:46.895069 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:18:48.059639 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:18:48.059597 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq"] Nov 03 14:18:48.106543 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:18:48.106506 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq"] Nov 03 14:19:01.108408 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:01.108370 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f"] Nov 03 14:19:01.216190 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:01.216150 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq"] Nov 03 14:19:03.088043 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.087997 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" podUID="293e19ad-0cde-46c2-8039-49927287f258" containerName="main" containerID="cri-o://358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033" gracePeriod=45 Nov 03 14:19:03.131495 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.131444 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" podUID="824d587c-60d2-4cf2-9c31-37f8baf60308" containerName="main" containerID="cri-o://b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085" gracePeriod=45 Nov 03 14:19:03.402512 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.402481 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 14:19:03.427438 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.427410 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 14:19:03.561584 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.561547 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-dshm\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-dshm\") pod \"824d587c-60d2-4cf2-9c31-37f8baf60308\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " Nov 03 14:19:03.561809 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.561623 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824d587c-60d2-4cf2-9c31-37f8baf60308-tls-certs\" (UniqueName: \"kubernetes.io/secret/824d587c-60d2-4cf2-9c31-37f8baf60308-tls-certs\") pod \"824d587c-60d2-4cf2-9c31-37f8baf60308\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " Nov 03 14:19:03.561809 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.561678 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/293e19ad-0cde-46c2-8039-49927287f258-kube-api-access-5627f\" (UniqueName: \"kubernetes.io/projected/293e19ad-0cde-46c2-8039-49927287f258-kube-api-access-5627f\") pod \"293e19ad-0cde-46c2-8039-49927287f258\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " Nov 03 14:19:03.561809 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.561715 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-home\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-home\") pod \"824d587c-60d2-4cf2-9c31-37f8baf60308\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " Nov 03 14:19:03.561809 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.561753 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/293e19ad-0cde-46c2-8039-49927287f258-tls-certs\" (UniqueName: \"kubernetes.io/secret/293e19ad-0cde-46c2-8039-49927287f258-tls-certs\") pod \"293e19ad-0cde-46c2-8039-49927287f258\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " Nov 03 14:19:03.562051 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.561856 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-tmp-dir\") pod \"824d587c-60d2-4cf2-9c31-37f8baf60308\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " Nov 03 14:19:03.562051 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.561888 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-model-cache\") pod \"293e19ad-0cde-46c2-8039-49927287f258\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " Nov 03 14:19:03.562159 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.562037 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-home" pod "824d587c-60d2-4cf2-9c31-37f8baf60308" (UID: "824d587c-60d2-4cf2-9c31-37f8baf60308"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.562243 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.562213 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-tmp-dir" pod "824d587c-60d2-4cf2-9c31-37f8baf60308" (UID: "824d587c-60d2-4cf2-9c31-37f8baf60308"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.562305 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.562231 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-model-cache" pod "293e19ad-0cde-46c2-8039-49927287f258" (UID: "293e19ad-0cde-46c2-8039-49927287f258"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.562362 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.562322 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824d587c-60d2-4cf2-9c31-37f8baf60308-kube-api-access-cflbk\" (UniqueName: \"kubernetes.io/projected/824d587c-60d2-4cf2-9c31-37f8baf60308-kube-api-access-cflbk\") pod \"824d587c-60d2-4cf2-9c31-37f8baf60308\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " Nov 03 14:19:03.562700 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.562676 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-tmp-dir\") pod \"293e19ad-0cde-46c2-8039-49927287f258\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " Nov 03 14:19:03.562988 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.562961 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-tmp-dir" pod "293e19ad-0cde-46c2-8039-49927287f258" (UID: "293e19ad-0cde-46c2-8039-49927287f258"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.563101 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.563083 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-home\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-home\") pod \"293e19ad-0cde-46c2-8039-49927287f258\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " Nov 03 14:19:03.563323 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.563301 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-home" pod "293e19ad-0cde-46c2-8039-49927287f258" (UID: "293e19ad-0cde-46c2-8039-49927287f258"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.563445 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.563426 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-model-cache\") pod \"824d587c-60d2-4cf2-9c31-37f8baf60308\" (UID: \"824d587c-60d2-4cf2-9c31-37f8baf60308\") " Nov 03 14:19:03.563686 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.563653 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-model-cache" pod "824d587c-60d2-4cf2-9c31-37f8baf60308" (UID: "824d587c-60d2-4cf2-9c31-37f8baf60308"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.563809 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.563710 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-dshm\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-dshm\") pod \"293e19ad-0cde-46c2-8039-49927287f258\" (UID: \"293e19ad-0cde-46c2-8039-49927287f258\") " Nov 03 14:19:03.564470 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564445 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824d587c-60d2-4cf2-9c31-37f8baf60308-tls-certs" pod "824d587c-60d2-4cf2-9c31-37f8baf60308" (UID: "824d587c-60d2-4cf2-9c31-37f8baf60308"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:03.564569 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564468 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-dshm" pod "824d587c-60d2-4cf2-9c31-37f8baf60308" (UID: "824d587c-60d2-4cf2-9c31-37f8baf60308"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.564629 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564566 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/293e19ad-0cde-46c2-8039-49927287f258-kube-api-access-5627f" pod "293e19ad-0cde-46c2-8039-49927287f258" (UID: "293e19ad-0cde-46c2-8039-49927287f258"). InnerVolumeSpecName "kube-api-access-5627f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:03.564629 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564610 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.564747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564630 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.564747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564643 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.564747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564655 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.564747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564668 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824d587c-60d2-4cf2-9c31-37f8baf60308-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.564747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564675 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.564747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564683 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824d587c-60d2-4cf2-9c31-37f8baf60308-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.564747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564691 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.565131 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.564902 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/293e19ad-0cde-46c2-8039-49927287f258-tls-certs" pod "293e19ad-0cde-46c2-8039-49927287f258" (UID: "293e19ad-0cde-46c2-8039-49927287f258"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:03.566482 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.566459 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824d587c-60d2-4cf2-9c31-37f8baf60308-kube-api-access-cflbk" pod "824d587c-60d2-4cf2-9c31-37f8baf60308" (UID: "824d587c-60d2-4cf2-9c31-37f8baf60308"). InnerVolumeSpecName "kube-api-access-cflbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:03.566889 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.566868 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-dshm" pod "293e19ad-0cde-46c2-8039-49927287f258" (UID: "293e19ad-0cde-46c2-8039-49927287f258"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.665967 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.665928 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cflbk\" (UniqueName: \"kubernetes.io/projected/824d587c-60d2-4cf2-9c31-37f8baf60308-kube-api-access-cflbk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.665967 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.665968 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/293e19ad-0cde-46c2-8039-49927287f258-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.666181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.665984 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5627f\" (UniqueName: \"kubernetes.io/projected/293e19ad-0cde-46c2-8039-49927287f258-kube-api-access-5627f\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.666181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:03.666002 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/293e19ad-0cde-46c2-8039-49927287f258-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:04.168913 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.168874 4541 generic.go:356] "Generic (PLEG): container finished" podID="293e19ad-0cde-46c2-8039-49927287f258" containerID="358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033" exitCode=0 Nov 03 14:19:04.169372 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.168917 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" Nov 03 14:19:04.169372 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.168956 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" event={"ID":"293e19ad-0cde-46c2-8039-49927287f258","Type":"ContainerDied","Data":"358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033"} Nov 03 14:19:04.169372 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.169005 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq" event={"ID":"293e19ad-0cde-46c2-8039-49927287f258","Type":"ContainerDied","Data":"6b994fd98906580a6a58b8db7819d7c4f0867ca6ee29edcf71b3d8a0313cf150"} Nov 03 14:19:04.169372 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.169026 4541 scope.go:122] "RemoveContainer" containerID="358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033" Nov 03 14:19:04.170365 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.170331 4541 generic.go:356] "Generic (PLEG): container finished" podID="824d587c-60d2-4cf2-9c31-37f8baf60308" containerID="b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085" exitCode=0 Nov 03 14:19:04.170454 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.170380 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" Nov 03 14:19:04.170454 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.170421 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" event={"ID":"824d587c-60d2-4cf2-9c31-37f8baf60308","Type":"ContainerDied","Data":"b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085"} Nov 03 14:19:04.170557 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.170471 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq" event={"ID":"824d587c-60d2-4cf2-9c31-37f8baf60308","Type":"ContainerDied","Data":"b2c208addb2bb4e95e5376bf125aaee11d15a8b423dd70e7359f3c72bb919bfb"} Nov 03 14:19:04.178996 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.178977 4541 scope.go:122] "RemoveContainer" containerID="358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033" Nov 03 14:19:04.179288 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:19:04.179267 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033\": container with ID starting with 358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033 not found: ID does not exist" containerID="358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033" Nov 03 14:19:04.179333 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.179297 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033"} err="failed to get container status \"358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033\": rpc error: code = NotFound desc = could not find container \"358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033\": container with ID starting with 358da0130e93879c6f0082a19ebde9f3c772a001eee761d9c5f3ff3f8d2bb033 not found: ID does not exist" Nov 03 14:19:04.179333 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.179315 4541 scope.go:122] "RemoveContainer" containerID="b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085" Nov 03 14:19:04.188834 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.188814 4541 scope.go:122] "RemoveContainer" containerID="b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085" Nov 03 14:19:04.189089 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:19:04.189070 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085\": container with ID starting with b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085 not found: ID does not exist" containerID="b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085" Nov 03 14:19:04.189153 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.189096 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085"} err="failed to get container status \"b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085\": rpc error: code = NotFound desc = could not find container \"b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085\": container with ID starting with b355605918cb46d819eb3d92df23f80997ffaddc954555d2a394ea927a24f085 not found: ID does not exist" Nov 03 14:19:04.328631 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.328597 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq"] Nov 03 14:19:04.375109 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.375032 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/simulator-w3-5bb463-kserve-b8bff58fb-dwvqq"] Nov 03 14:19:04.439466 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.439425 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq"] Nov 03 14:19:04.444498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.444470 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-w3-5bb463-kserve-5448c6dc6d-fdkbq"] Nov 03 14:19:04.874797 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.870252 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="293e19ad-0cde-46c2-8039-49927287f258" path="/var/lib/kubelet/pods/293e19ad-0cde-46c2-8039-49927287f258/volumes" Nov 03 14:19:04.874797 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:04.871044 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824d587c-60d2-4cf2-9c31-37f8baf60308" path="/var/lib/kubelet/pods/824d587c-60d2-4cf2-9c31-37f8baf60308/volumes" Nov 03 14:19:06.109312 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:06.109264 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" podUID="79f72dbc-5398-42cd-95b0-1f70910389b2" containerName="payload-processing" containerID="cri-o://e823d41ea91619745b09a12c4bc60455e578165f96a908feb0b3aa9b58e2d396" gracePeriod=25 Nov 03 14:19:06.217323 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:06.217278 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" podUID="e528f67e-a234-452d-8c81-3e8f2f0d31ac" containerName="payload-pre-processing" containerID="cri-o://171cee89fd955515495805c6edb90cfc943a4dd1fcf346c3094de2b963eafac8" gracePeriod=25 Nov 03 14:19:07.202026 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:07.201986 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" podUID="e528f67e-a234-452d-8c81-3e8f2f0d31ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:19:07.202454 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:07.202002 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" podUID="79f72dbc-5398-42cd-95b0-1f70910389b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:19:15.396498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.396458 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz"] Nov 03 14:19:15.397172 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397144 4541 cpu_manager.go:414] "containerMap: removing container" podUID="60097229-aca6-41fc-82e7-9f523a144f87" containerName="cleanup" Nov 03 14:19:15.397172 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397170 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60097229-aca6-41fc-82e7-9f523a144f87" containerName="cleanup" Nov 03 14:19:15.397367 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397228 4541 cpu_manager.go:414] "containerMap: removing container" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerName="cleanup" Nov 03 14:19:15.397367 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397262 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerName="cleanup" Nov 03 14:19:15.397367 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397280 4541 cpu_manager.go:414] "containerMap: removing container" podUID="697c9c9c-206e-44c2-8ded-4e10d15a8ad2" containerName="payload-pre-processing" Nov 03 14:19:15.397367 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397290 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="697c9c9c-206e-44c2-8ded-4e10d15a8ad2" containerName="payload-pre-processing" Nov 03 14:19:15.397367 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397328 4541 cpu_manager.go:414] "containerMap: removing container" podUID="293e19ad-0cde-46c2-8039-49927287f258" containerName="main" Nov 03 14:19:15.397367 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397336 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="293e19ad-0cde-46c2-8039-49927287f258" containerName="main" Nov 03 14:19:15.397367 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397364 4541 cpu_manager.go:414] "containerMap: removing container" podUID="824d587c-60d2-4cf2-9c31-37f8baf60308" containerName="main" Nov 03 14:19:15.397695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397374 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824d587c-60d2-4cf2-9c31-37f8baf60308" containerName="main" Nov 03 14:19:15.397695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397524 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="293e19ad-0cde-46c2-8039-49927287f258" containerName="main" Nov 03 14:19:15.397695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397537 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="60097229-aca6-41fc-82e7-9f523a144f87" containerName="cleanup" Nov 03 14:19:15.397695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397545 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="824d587c-60d2-4cf2-9c31-37f8baf60308" containerName="main" Nov 03 14:19:15.397695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397550 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f151f83-66bc-4d9f-ab3d-29fb143ad893" containerName="cleanup" Nov 03 14:19:15.397695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.397565 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="697c9c9c-206e-44c2-8ded-4e10d15a8ad2" containerName="payload-pre-processing" Nov 03 14:19:15.400996 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.400979 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.498250 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.498208 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"rate-test-model-2bceb1-kserve-self-signed-certs\"" Nov 03 14:19:15.498250 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.498220 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"default-dockercfg-rxrxl\"" Nov 03 14:19:15.498250 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.498220 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"openshift-service-ca.crt\"" Nov 03 14:19:15.498566 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.498245 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2bceb1\"/\"kube-root-ca.crt\"" Nov 03 14:19:15.501237 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.501215 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz"] Nov 03 14:19:15.504803 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.504766 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-dshm\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.504908 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.504889 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-home\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.504992 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.504965 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tmp-dir\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.505042 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.505003 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tls-certs\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.505134 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.505113 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-model-cache\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.505193 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.505160 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rprk6\" (UniqueName: \"kubernetes.io/projected/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-kube-api-access-rprk6\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.606158 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.606121 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tmp-dir\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.606319 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.606176 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tls-certs\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.606319 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.606225 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-model-cache\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.606319 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.606245 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rprk6\" (UniqueName: \"kubernetes.io/projected/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-kube-api-access-rprk6\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.606319 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.606266 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-dshm\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.606561 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.606345 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-home\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.606635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.606607 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tmp-dir\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.606703 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.606652 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-model-cache\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.606752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.606740 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-home\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.608956 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.608923 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-dshm\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.609148 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.609127 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tls-certs\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.661339 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.661306 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rprk6\" (UniqueName: \"kubernetes.io/projected/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-kube-api-access-rprk6\") pod \"rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:15.713066 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:15.713025 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:16.083336 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:16.083307 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz"] Nov 03 14:19:16.084516 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:19:16.084486 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb04af2a4_f3f3_41ee_a562_1c66dc4bc8b5.slice/crio-fd1ea7bce616f4fe8792ddaf56c08e11deefc14892e4e1f7639c230a736e8abb WatchSource:0}: Error finding container fd1ea7bce616f4fe8792ddaf56c08e11deefc14892e4e1f7639c230a736e8abb: Status 404 returned error can't find the container with id fd1ea7bce616f4fe8792ddaf56c08e11deefc14892e4e1f7639c230a736e8abb Nov 03 14:19:16.252629 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:16.252595 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" event={"ID":"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5","Type":"ContainerStarted","Data":"18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a"} Nov 03 14:19:16.252830 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:16.252641 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" event={"ID":"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5","Type":"ContainerStarted","Data":"fd1ea7bce616f4fe8792ddaf56c08e11deefc14892e4e1f7639c230a736e8abb"} Nov 03 14:19:16.363870 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:16.363739 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" podStartSLOduration=1.3637252260000001 podStartE2EDuration="1.363725226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:16.361492051 +0000 UTC m=+4487.675463911" watchObservedRunningTime="2026-11-03 14:19:16.363725226 +0000 UTC m=+4487.677697149" Nov 03 14:19:16.713865 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:16.713826 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:16.894206 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:19:16.894167 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:19:17.202584 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:17.202540 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" podUID="79f72dbc-5398-42cd-95b0-1f70910389b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:19:17.203108 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:17.203080 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" podUID="e528f67e-a234-452d-8c81-3e8f2f0d31ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:19:23.627923 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.627879 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk"] Nov 03 14:19:23.632230 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.632203 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.728621 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.728584 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b5daee\"/\"openshift-service-ca.crt\"" Nov 03 14:19:23.728882 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.728861 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b5daee\"/\"kube-root-ca.crt\"" Nov 03 14:19:23.728991 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.728875 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b5daee\"/\"test-model-b5daee-kserve-self-signed-certs\"" Nov 03 14:19:23.728991 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.728905 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b5daee\"/\"default-dockercfg-6zdbd\"" Nov 03 14:19:23.731753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.731730 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk"] Nov 03 14:19:23.755438 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.755403 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-home\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.755438 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.755439 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tls-certs\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.755689 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.755600 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skb55\" (UniqueName: \"kubernetes.io/projected/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-kube-api-access-skb55\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.755753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.755698 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-model-cache\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.755853 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.755830 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-dshm\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.755924 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.755908 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tmp-dir\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.856853 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.856816 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-home\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.856853 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.856861 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tls-certs\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.857124 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.856934 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skb55\" (UniqueName: \"kubernetes.io/projected/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-kube-api-access-skb55\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.857124 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.857011 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-model-cache\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.857124 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.857109 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-dshm\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.857288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.857163 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tmp-dir\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.857349 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.857321 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-home\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.857405 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.857375 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-model-cache\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.857552 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.857533 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tmp-dir\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.859345 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.859318 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-dshm\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.859750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.859728 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tls-certs\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.911435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.911400 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skb55\" (UniqueName: \"kubernetes.io/projected/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-kube-api-access-skb55\") pod \"test-model-b5daee-kserve-66c67dfc55-qxssk\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:23.944994 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:23.944956 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:24.317571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:24.317541 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk"] Nov 03 14:19:24.318665 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:19:24.318631 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded40d4b9_4b0f_4500_ae39_1f6b12d49d98.slice/crio-87476e9863abe19fa368d751472c50ff568dd2fbada3a5a3f99f6d219a4270ab WatchSource:0}: Error finding container 87476e9863abe19fa368d751472c50ff568dd2fbada3a5a3f99f6d219a4270ab: Status 404 returned error can't find the container with id 87476e9863abe19fa368d751472c50ff568dd2fbada3a5a3f99f6d219a4270ab Nov 03 14:19:24.348905 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:24.348866 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" event={"ID":"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98","Type":"ContainerStarted","Data":"87476e9863abe19fa368d751472c50ff568dd2fbada3a5a3f99f6d219a4270ab"} Nov 03 14:19:25.356986 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:25.356946 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" event={"ID":"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98","Type":"ContainerStarted","Data":"d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a"} Nov 03 14:19:25.467964 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:25.467911 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" podStartSLOduration=2.46789307 podStartE2EDuration="2.46789307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:19:25.467437531 +0000 UTC m=+4496.781409390" watchObservedRunningTime="2026-11-03 14:19:25.46789307 +0000 UTC m=+4496.781864931" Nov 03 14:19:25.713536 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:25.713498 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:25.726675 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:25.726650 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:25.945828 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:25.945788 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:26.367291 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.367215 4541 generic.go:356] "Generic (PLEG): container finished" podID="79f72dbc-5398-42cd-95b0-1f70910389b2" containerID="e823d41ea91619745b09a12c4bc60455e578165f96a908feb0b3aa9b58e2d396" exitCode=0 Nov 03 14:19:26.367686 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.367362 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" event={"ID":"79f72dbc-5398-42cd-95b0-1f70910389b2","Type":"ContainerDied","Data":"e823d41ea91619745b09a12c4bc60455e578165f96a908feb0b3aa9b58e2d396"} Nov 03 14:19:26.370799 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.370749 4541 generic.go:356] "Generic (PLEG): container finished" podID="e528f67e-a234-452d-8c81-3e8f2f0d31ac" containerID="171cee89fd955515495805c6edb90cfc943a4dd1fcf346c3094de2b963eafac8" exitCode=0 Nov 03 14:19:26.370930 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.370820 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" event={"ID":"e528f67e-a234-452d-8c81-3e8f2f0d31ac","Type":"ContainerDied","Data":"171cee89fd955515495805c6edb90cfc943a4dd1fcf346c3094de2b963eafac8"} Nov 03 14:19:26.387631 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.387608 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:19:26.393640 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.393616 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 14:19:26.493060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.493024 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f72dbc-5398-42cd-95b0-1f70910389b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f72dbc-5398-42cd-95b0-1f70910389b2-plugins-config-volume\") pod \"79f72dbc-5398-42cd-95b0-1f70910389b2\" (UID: \"79f72dbc-5398-42cd-95b0-1f70910389b2\") " Nov 03 14:19:26.493281 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.493256 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f72dbc-5398-42cd-95b0-1f70910389b2-kube-api-access-5lg5f\" (UniqueName: \"kubernetes.io/projected/79f72dbc-5398-42cd-95b0-1f70910389b2-kube-api-access-5lg5f\") pod \"79f72dbc-5398-42cd-95b0-1f70910389b2\" (UID: \"79f72dbc-5398-42cd-95b0-1f70910389b2\") " Nov 03 14:19:26.493520 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.493489 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f72dbc-5398-42cd-95b0-1f70910389b2-plugins-config-volume" pod "79f72dbc-5398-42cd-95b0-1f70910389b2" (UID: "79f72dbc-5398-42cd-95b0-1f70910389b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:19:26.493812 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.493791 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f72dbc-5398-42cd-95b0-1f70910389b2-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:26.496283 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.496247 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f72dbc-5398-42cd-95b0-1f70910389b2-kube-api-access-5lg5f" pod "79f72dbc-5398-42cd-95b0-1f70910389b2" (UID: "79f72dbc-5398-42cd-95b0-1f70910389b2"). InnerVolumeSpecName "kube-api-access-5lg5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:26.516718 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.516686 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 14:19:26.594994 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.594961 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e528f67e-a234-452d-8c81-3e8f2f0d31ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e528f67e-a234-452d-8c81-3e8f2f0d31ac-plugins-config-volume\") pod \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\" (UID: \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\") " Nov 03 14:19:26.595180 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.595089 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e528f67e-a234-452d-8c81-3e8f2f0d31ac-kube-api-access-qd9vl\" (UniqueName: \"kubernetes.io/projected/e528f67e-a234-452d-8c81-3e8f2f0d31ac-kube-api-access-qd9vl\") pod \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\" (UID: \"e528f67e-a234-452d-8c81-3e8f2f0d31ac\") " Nov 03 14:19:26.595369 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.595343 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e528f67e-a234-452d-8c81-3e8f2f0d31ac-plugins-config-volume" pod "e528f67e-a234-452d-8c81-3e8f2f0d31ac" (UID: "e528f67e-a234-452d-8c81-3e8f2f0d31ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:19:26.595563 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.595547 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e528f67e-a234-452d-8c81-3e8f2f0d31ac-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:26.595630 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.595569 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lg5f\" (UniqueName: \"kubernetes.io/projected/79f72dbc-5398-42cd-95b0-1f70910389b2-kube-api-access-5lg5f\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:26.597169 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.597141 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e528f67e-a234-452d-8c81-3e8f2f0d31ac-kube-api-access-qd9vl" pod "e528f67e-a234-452d-8c81-3e8f2f0d31ac" (UID: "e528f67e-a234-452d-8c81-3e8f2f0d31ac"). InnerVolumeSpecName "kube-api-access-qd9vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:26.696592 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:26.696550 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd9vl\" (UniqueName: \"kubernetes.io/projected/e528f67e-a234-452d-8c81-3e8f2f0d31ac-kube-api-access-qd9vl\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:27.378698 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.378619 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" Nov 03 14:19:27.379183 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.378932 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f" event={"ID":"79f72dbc-5398-42cd-95b0-1f70910389b2","Type":"ContainerDied","Data":"fbe5040ab15f8432aa9de6d4bf50924bb461f5b59342a41cdd0c46f24f7fbfbc"} Nov 03 14:19:27.379183 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.379010 4541 scope.go:122] "RemoveContainer" containerID="e823d41ea91619745b09a12c4bc60455e578165f96a908feb0b3aa9b58e2d396" Nov 03 14:19:27.380432 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.380413 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" Nov 03 14:19:27.380512 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.380449 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq" event={"ID":"e528f67e-a234-452d-8c81-3e8f2f0d31ac","Type":"ContainerDied","Data":"42c26712288fbaffae1f195300938c4645109a044615112e30bf018c6a2ceba5"} Nov 03 14:19:27.390935 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.390917 4541 scope.go:122] "RemoveContainer" containerID="171cee89fd955515495805c6edb90cfc943a4dd1fcf346c3094de2b963eafac8" Nov 03 14:19:27.534230 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.534198 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq"] Nov 03 14:19:27.542283 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.542255 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-5bb463-65776ff89f-rlpfq"] Nov 03 14:19:27.683000 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.682964 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f"] Nov 03 14:19:27.686715 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:27.686689 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-5bb463-5bc85cc84f-t7c4f"] Nov 03 14:19:28.870287 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:28.870253 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f72dbc-5398-42cd-95b0-1f70910389b2" path="/var/lib/kubelet/pods/79f72dbc-5398-42cd-95b0-1f70910389b2/volumes" Nov 03 14:19:28.870678 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:28.870664 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e528f67e-a234-452d-8c81-3e8f2f0d31ac" path="/var/lib/kubelet/pods/e528f67e-a234-452d-8c81-3e8f2f0d31ac/volumes" Nov 03 14:19:29.438238 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:29.438204 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:19:29.452742 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:29.452716 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:19:33.945873 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:33.945838 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:33.958480 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:33.958453 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:34.439927 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:34.439895 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:19:36.141801 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.141751 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz"] Nov 03 14:19:36.142181 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.142037 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" podUID="5c103702-fbda-41d3-bf5a-86933fac7f40" containerName="maas-api" containerID="cri-o://dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3" gracePeriod=30 Nov 03 14:19:36.283709 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.283661 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" podUID="5c103702-fbda-41d3-bf5a-86933fac7f40" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.56:8443/health\": dial tcp 10.133.0.56:8443: connect: connection refused" Nov 03 14:19:36.411673 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.411649 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 14:19:36.438520 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.438483 4541 generic.go:356] "Generic (PLEG): container finished" podID="5c103702-fbda-41d3-bf5a-86933fac7f40" containerID="dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3" exitCode=0 Nov 03 14:19:36.438654 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.438556 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" Nov 03 14:19:36.438654 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.438569 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" event={"ID":"5c103702-fbda-41d3-bf5a-86933fac7f40","Type":"ContainerDied","Data":"dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3"} Nov 03 14:19:36.438654 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.438629 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz" event={"ID":"5c103702-fbda-41d3-bf5a-86933fac7f40","Type":"ContainerDied","Data":"51ddc3ec423b355ae03683caae91709734b20cb37be174c7a3d4b1a16da70067"} Nov 03 14:19:36.438814 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.438657 4541 scope.go:122] "RemoveContainer" containerID="dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3" Nov 03 14:19:36.448508 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.448489 4541 scope.go:122] "RemoveContainer" containerID="dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3" Nov 03 14:19:36.448805 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:19:36.448780 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3\": container with ID starting with dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3 not found: ID does not exist" containerID="dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3" Nov 03 14:19:36.448862 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.448812 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3"} err="failed to get container status \"dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3\": rpc error: code = NotFound desc = could not find container \"dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3\": container with ID starting with dad763fc1c8747a684ac03514be5d33e846d34d931d7fc69859902943f6ddaa3 not found: ID does not exist" Nov 03 14:19:36.507155 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.507128 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-maas-api-tls\") pod \"5c103702-fbda-41d3-bf5a-86933fac7f40\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " Nov 03 14:19:36.507268 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.507230 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c103702-fbda-41d3-bf5a-86933fac7f40-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c103702-fbda-41d3-bf5a-86933fac7f40-openshift-service-ca\") pod \"5c103702-fbda-41d3-bf5a-86933fac7f40\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " Nov 03 14:19:36.507311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.507273 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c103702-fbda-41d3-bf5a-86933fac7f40-kube-api-access-c6sww\" (UniqueName: \"kubernetes.io/projected/5c103702-fbda-41d3-bf5a-86933fac7f40-kube-api-access-c6sww\") pod \"5c103702-fbda-41d3-bf5a-86933fac7f40\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " Nov 03 14:19:36.507349 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.507325 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-metrics-certs\") pod \"5c103702-fbda-41d3-bf5a-86933fac7f40\" (UID: \"5c103702-fbda-41d3-bf5a-86933fac7f40\") " Nov 03 14:19:36.507618 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.507595 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c103702-fbda-41d3-bf5a-86933fac7f40-openshift-service-ca" pod "5c103702-fbda-41d3-bf5a-86933fac7f40" (UID: "5c103702-fbda-41d3-bf5a-86933fac7f40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:19:36.507873 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.507757 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c103702-fbda-41d3-bf5a-86933fac7f40-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:36.509257 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.509232 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-maas-api-tls" pod "5c103702-fbda-41d3-bf5a-86933fac7f40" (UID: "5c103702-fbda-41d3-bf5a-86933fac7f40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:36.509368 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.509348 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c103702-fbda-41d3-bf5a-86933fac7f40-kube-api-access-c6sww" pod "5c103702-fbda-41d3-bf5a-86933fac7f40" (UID: "5c103702-fbda-41d3-bf5a-86933fac7f40"). InnerVolumeSpecName "kube-api-access-c6sww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:36.509551 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.509519 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-metrics-certs" pod "5c103702-fbda-41d3-bf5a-86933fac7f40" (UID: "5c103702-fbda-41d3-bf5a-86933fac7f40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:36.608788 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.608748 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6sww\" (UniqueName: \"kubernetes.io/projected/5c103702-fbda-41d3-bf5a-86933fac7f40-kube-api-access-c6sww\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:36.608960 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.608796 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:36.608960 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.608813 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c103702-fbda-41d3-bf5a-86933fac7f40-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:36.853411 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.853370 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz"] Nov 03 14:19:36.857062 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.857035 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-5bb463-fd4c885b5-fmpbz"] Nov 03 14:19:36.868161 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:19:36.868130 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c103702-fbda-41d3-bf5a-86933fac7f40" path="/var/lib/kubelet/pods/5c103702-fbda-41d3-bf5a-86933fac7f40/volumes" Nov 03 14:19:46.893740 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:19:46.893703 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:20:16.891396 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:20:16.891358 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:20:46.891828 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:20:46.891785 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:20:58.468146 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468106 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv"] Nov 03 14:20:58.468682 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468667 4541 cpu_manager.go:414] "containerMap: removing container" podUID="79f72dbc-5398-42cd-95b0-1f70910389b2" containerName="payload-processing" Nov 03 14:20:58.468730 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468683 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f72dbc-5398-42cd-95b0-1f70910389b2" containerName="payload-processing" Nov 03 14:20:58.468730 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468705 4541 cpu_manager.go:414] "containerMap: removing container" podUID="e528f67e-a234-452d-8c81-3e8f2f0d31ac" containerName="payload-pre-processing" Nov 03 14:20:58.468730 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468711 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e528f67e-a234-452d-8c81-3e8f2f0d31ac" containerName="payload-pre-processing" Nov 03 14:20:58.468730 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468718 4541 cpu_manager.go:414] "containerMap: removing container" podUID="5c103702-fbda-41d3-bf5a-86933fac7f40" containerName="maas-api" Nov 03 14:20:58.468730 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468724 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c103702-fbda-41d3-bf5a-86933fac7f40" containerName="maas-api" Nov 03 14:20:58.468936 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468822 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f72dbc-5398-42cd-95b0-1f70910389b2" containerName="payload-processing" Nov 03 14:20:58.468936 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468834 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="e528f67e-a234-452d-8c81-3e8f2f0d31ac" containerName="payload-pre-processing" Nov 03 14:20:58.468936 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.468842 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c103702-fbda-41d3-bf5a-86933fac7f40" containerName="maas-api" Nov 03 14:20:58.472245 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.472225 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.552944 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.552907 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttzdv\" (UniqueName: \"kubernetes.io/projected/948aef81-0234-4290-897f-bba6242b630e-kube-api-access-ttzdv\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.553138 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.553080 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948aef81-0234-4290-897f-bba6242b630e-tls-certs\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.553138 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.553126 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-home\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.553268 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.553253 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-tmp-dir\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.553426 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.553408 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-model-cache\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.553518 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.553492 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-dshm\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.569494 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.569464 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"kube-root-ca.crt\"" Nov 03 14:20:58.569667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.569526 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"rate-test-model-d341af-kserve-self-signed-certs\"" Nov 03 14:20:58.569667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.569530 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"default-dockercfg-hln72\"" Nov 03 14:20:58.572928 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.572908 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv"] Nov 03 14:20:58.623248 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.623211 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-d341af\"/\"openshift-service-ca.crt\"" Nov 03 14:20:58.655138 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.655099 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-model-cache\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.655138 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.655144 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-dshm\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.655363 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.655166 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttzdv\" (UniqueName: \"kubernetes.io/projected/948aef81-0234-4290-897f-bba6242b630e-kube-api-access-ttzdv\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.655363 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.655204 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948aef81-0234-4290-897f-bba6242b630e-tls-certs\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.655363 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.655231 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-home\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.655363 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.655256 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-tmp-dir\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.655628 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.655602 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-model-cache\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.655628 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.655620 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-tmp-dir\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.655797 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.655701 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-home\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.657512 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.657493 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-dshm\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.657681 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.657663 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948aef81-0234-4290-897f-bba6242b630e-tls-certs\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.709265 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.709229 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttzdv\" (UniqueName: \"kubernetes.io/projected/948aef81-0234-4290-897f-bba6242b630e-kube-api-access-ttzdv\") pod \"rate-test-model-d341af-kserve-546c59794f-5x5hv\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:58.784506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:58.784419 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:20:59.162779 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:59.162728 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv"] Nov 03 14:20:59.934578 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:59.934544 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" event={"ID":"948aef81-0234-4290-897f-bba6242b630e","Type":"ContainerStarted","Data":"14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067"} Nov 03 14:20:59.935071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:20:59.934588 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" event={"ID":"948aef81-0234-4290-897f-bba6242b630e","Type":"ContainerStarted","Data":"bbe04203bf5c2d9c33e3221179fe8edf339a36b24ca209679435d30228a8be1f"} Nov 03 14:21:00.047148 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:21:00.047092 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" podStartSLOduration=2.047077975 podStartE2EDuration="2.047077975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:21:00.045621998 +0000 UTC m=+4591.359593867" watchObservedRunningTime="2026-11-03 14:21:00.047077975 +0000 UTC m=+4591.361049868" Nov 03 14:21:00.785457 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:21:00.785414 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:21:08.784992 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:21:08.784955 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:21:08.797581 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:21:08.797555 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:21:08.998877 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:21:08.998844 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:21:16.890023 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:21:16.889983 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:21:46.891250 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:21:46.891204 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:22:16.889658 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:22:16.889605 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:22:24.910765 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:24.910726 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc"] Nov 03 14:22:29.911744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:29.911696 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" podUID="dcea7270-5708-4b70-b0dd-15304574e276" containerName="payload-processing" containerID="cri-o://83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121" gracePeriod=25 Nov 03 14:22:31.427948 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:31.427909 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" podUID="dcea7270-5708-4b70-b0dd-15304574e276" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:22:41.427455 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:41.427412 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" podUID="dcea7270-5708-4b70-b0dd-15304574e276" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:22:46.891573 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:22:46.891523 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:22:50.185506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.185438 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 14:22:50.213948 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.213915 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcea7270-5708-4b70-b0dd-15304574e276-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea7270-5708-4b70-b0dd-15304574e276-plugins-config-volume\") pod \"dcea7270-5708-4b70-b0dd-15304574e276\" (UID: \"dcea7270-5708-4b70-b0dd-15304574e276\") " Nov 03 14:22:50.214098 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.214005 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcea7270-5708-4b70-b0dd-15304574e276-kube-api-access-l7kr7\" (UniqueName: \"kubernetes.io/projected/dcea7270-5708-4b70-b0dd-15304574e276-kube-api-access-l7kr7\") pod \"dcea7270-5708-4b70-b0dd-15304574e276\" (UID: \"dcea7270-5708-4b70-b0dd-15304574e276\") " Nov 03 14:22:50.214381 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.214343 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcea7270-5708-4b70-b0dd-15304574e276-plugins-config-volume" pod "dcea7270-5708-4b70-b0dd-15304574e276" (UID: "dcea7270-5708-4b70-b0dd-15304574e276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:22:50.216189 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.216160 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcea7270-5708-4b70-b0dd-15304574e276-kube-api-access-l7kr7" pod "dcea7270-5708-4b70-b0dd-15304574e276" (UID: "dcea7270-5708-4b70-b0dd-15304574e276"). InnerVolumeSpecName "kube-api-access-l7kr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:22:50.315870 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.315836 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcea7270-5708-4b70-b0dd-15304574e276-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:50.315870 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.315866 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7kr7\" (UniqueName: \"kubernetes.io/projected/dcea7270-5708-4b70-b0dd-15304574e276-kube-api-access-l7kr7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:50.614514 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.614424 4541 generic.go:356] "Generic (PLEG): container finished" podID="dcea7270-5708-4b70-b0dd-15304574e276" containerID="83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121" exitCode=0 Nov 03 14:22:50.614514 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.614477 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" Nov 03 14:22:50.614697 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.614492 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" event={"ID":"dcea7270-5708-4b70-b0dd-15304574e276","Type":"ContainerDied","Data":"83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121"} Nov 03 14:22:50.614697 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.614603 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc" event={"ID":"dcea7270-5708-4b70-b0dd-15304574e276","Type":"ContainerDied","Data":"c5ff9a7121d7718f4eb1ad6616c1345df78fb91acd9eb6ce4dbcf89049be436c"} Nov 03 14:22:50.614697 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.614629 4541 scope.go:122] "RemoveContainer" containerID="83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121" Nov 03 14:22:50.624716 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.624695 4541 scope.go:122] "RemoveContainer" containerID="83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121" Nov 03 14:22:50.625015 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:22:50.624994 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121\": container with ID starting with 83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121 not found: ID does not exist" containerID="83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121" Nov 03 14:22:50.625073 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.625023 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121"} err="failed to get container status \"83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121\": rpc error: code = NotFound desc = could not find container \"83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121\": container with ID starting with 83adee6352bd2085acc7893fe5b3791f767fd8dded2555d804bc08e8a97a6121 not found: ID does not exist" Nov 03 14:22:50.769137 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.769096 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc"] Nov 03 14:22:50.773507 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.773474 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-5faf5d-8b8c9bccf-mpjrc"] Nov 03 14:22:50.868940 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:22:50.868865 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcea7270-5708-4b70-b0dd-15304574e276" path="/var/lib/kubelet/pods/dcea7270-5708-4b70-b0dd-15304574e276/volumes" Nov 03 14:23:16.897607 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:23:16.897521 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:23:39.201875 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:23:39.201841 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7"] Nov 03 14:23:39.311401 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:23:39.311366 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj"] Nov 03 14:23:44.202984 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:23:44.202923 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" podUID="256691b8-991d-4105-9b10-40dbcc6dcc2c" containerName="payload-processing" containerID="cri-o://6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95" gracePeriod=25 Nov 03 14:23:44.312238 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:23:44.312191 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" podUID="a01491e5-7f57-4419-9186-c2966e8cc22f" containerName="payload-pre-processing" containerID="cri-o://1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b" gracePeriod=25 Nov 03 14:23:46.890333 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:23:46.890291 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:23:49.284103 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:23:49.284057 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" podUID="a01491e5-7f57-4419-9186-c2966e8cc22f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:23:49.284552 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:23:49.284115 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" podUID="256691b8-991d-4105-9b10-40dbcc6dcc2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:23:59.283708 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:23:59.283665 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" podUID="a01491e5-7f57-4419-9186-c2966e8cc22f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:23:59.284227 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:23:59.283665 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" podUID="256691b8-991d-4105-9b10-40dbcc6dcc2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:24:04.508311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.508263 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 14:24:04.595005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.594980 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 14:24:04.681384 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.681352 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256691b8-991d-4105-9b10-40dbcc6dcc2c-kube-api-access-sk5s8\" (UniqueName: \"kubernetes.io/projected/256691b8-991d-4105-9b10-40dbcc6dcc2c-kube-api-access-sk5s8\") pod \"256691b8-991d-4105-9b10-40dbcc6dcc2c\" (UID: \"256691b8-991d-4105-9b10-40dbcc6dcc2c\") " Nov 03 14:24:04.681560 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.681455 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01491e5-7f57-4419-9186-c2966e8cc22f-kube-api-access-bzv5k\" (UniqueName: \"kubernetes.io/projected/a01491e5-7f57-4419-9186-c2966e8cc22f-kube-api-access-bzv5k\") pod \"a01491e5-7f57-4419-9186-c2966e8cc22f\" (UID: \"a01491e5-7f57-4419-9186-c2966e8cc22f\") " Nov 03 14:24:04.681560 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.681479 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01491e5-7f57-4419-9186-c2966e8cc22f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01491e5-7f57-4419-9186-c2966e8cc22f-plugins-config-volume\") pod \"a01491e5-7f57-4419-9186-c2966e8cc22f\" (UID: \"a01491e5-7f57-4419-9186-c2966e8cc22f\") " Nov 03 14:24:04.681656 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.681580 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256691b8-991d-4105-9b10-40dbcc6dcc2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256691b8-991d-4105-9b10-40dbcc6dcc2c-plugins-config-volume\") pod \"256691b8-991d-4105-9b10-40dbcc6dcc2c\" (UID: \"256691b8-991d-4105-9b10-40dbcc6dcc2c\") " Nov 03 14:24:04.682025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.681989 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01491e5-7f57-4419-9186-c2966e8cc22f-plugins-config-volume" pod "a01491e5-7f57-4419-9186-c2966e8cc22f" (UID: "a01491e5-7f57-4419-9186-c2966e8cc22f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:04.682025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.681996 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256691b8-991d-4105-9b10-40dbcc6dcc2c-plugins-config-volume" pod "256691b8-991d-4105-9b10-40dbcc6dcc2c" (UID: "256691b8-991d-4105-9b10-40dbcc6dcc2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:04.682197 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.682112 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a01491e5-7f57-4419-9186-c2966e8cc22f-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:04.682197 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.682129 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/256691b8-991d-4105-9b10-40dbcc6dcc2c-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:04.684099 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.684077 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01491e5-7f57-4419-9186-c2966e8cc22f-kube-api-access-bzv5k" pod "a01491e5-7f57-4419-9186-c2966e8cc22f" (UID: "a01491e5-7f57-4419-9186-c2966e8cc22f"). InnerVolumeSpecName "kube-api-access-bzv5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:04.684192 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.684105 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256691b8-991d-4105-9b10-40dbcc6dcc2c-kube-api-access-sk5s8" pod "256691b8-991d-4105-9b10-40dbcc6dcc2c" (UID: "256691b8-991d-4105-9b10-40dbcc6dcc2c"). InnerVolumeSpecName "kube-api-access-sk5s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:04.783442 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.783335 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk5s8\" (UniqueName: \"kubernetes.io/projected/256691b8-991d-4105-9b10-40dbcc6dcc2c-kube-api-access-sk5s8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:04.783442 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:04.783394 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzv5k\" (UniqueName: \"kubernetes.io/projected/a01491e5-7f57-4419-9186-c2966e8cc22f-kube-api-access-bzv5k\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:05.071742 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.071653 4541 generic.go:356] "Generic (PLEG): container finished" podID="256691b8-991d-4105-9b10-40dbcc6dcc2c" containerID="6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95" exitCode=0 Nov 03 14:24:05.071742 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.071689 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" event={"ID":"256691b8-991d-4105-9b10-40dbcc6dcc2c","Type":"ContainerDied","Data":"6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95"} Nov 03 14:24:05.071975 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.071746 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" event={"ID":"256691b8-991d-4105-9b10-40dbcc6dcc2c","Type":"ContainerDied","Data":"08a18980967c5c9a1ea43a7effb28ce97a9364dda00dcdb2bfea0050a97c9bb3"} Nov 03 14:24:05.071975 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.071747 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7" Nov 03 14:24:05.071975 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.071783 4541 scope.go:122] "RemoveContainer" containerID="6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95" Nov 03 14:24:05.077938 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.077907 4541 generic.go:356] "Generic (PLEG): container finished" podID="a01491e5-7f57-4419-9186-c2966e8cc22f" containerID="1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b" exitCode=0 Nov 03 14:24:05.078089 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.077986 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" event={"ID":"a01491e5-7f57-4419-9186-c2966e8cc22f","Type":"ContainerDied","Data":"1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b"} Nov 03 14:24:05.078089 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.078034 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" event={"ID":"a01491e5-7f57-4419-9186-c2966e8cc22f","Type":"ContainerDied","Data":"3b0ad39c79a0996c53d7b02452fb89910894a27cf1c3dc7529f013f99ef53c30"} Nov 03 14:24:05.078089 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.077992 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj" Nov 03 14:24:05.085570 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.085548 4541 scope.go:122] "RemoveContainer" containerID="6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95" Nov 03 14:24:05.085883 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:24:05.085865 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95\": container with ID starting with 6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95 not found: ID does not exist" containerID="6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95" Nov 03 14:24:05.085955 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.085891 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95"} err="failed to get container status \"6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95\": rpc error: code = NotFound desc = could not find container \"6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95\": container with ID starting with 6843bc97641cd2abefcc4bc58830b8ec33424381d55c073ce0893836811a2d95 not found: ID does not exist" Nov 03 14:24:05.085955 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.085909 4541 scope.go:122] "RemoveContainer" containerID="1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b" Nov 03 14:24:05.096058 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.096036 4541 scope.go:122] "RemoveContainer" containerID="1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b" Nov 03 14:24:05.096399 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:24:05.096371 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b\": container with ID starting with 1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b not found: ID does not exist" containerID="1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b" Nov 03 14:24:05.096493 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.096410 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b"} err="failed to get container status \"1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b\": rpc error: code = NotFound desc = could not find container \"1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b\": container with ID starting with 1a5d0d9265e19708163f5fb1ce369ab680094f1d81dd133413c7850913a5c82b not found: ID does not exist" Nov 03 14:24:05.230614 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.230579 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj"] Nov 03 14:24:05.278140 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.278098 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-d0231d-56f687cb79-xqdtj"] Nov 03 14:24:05.346365 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.346279 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7"] Nov 03 14:24:05.357938 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:05.357905 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-d0231d-5f7f8c76f9-7nfx7"] Nov 03 14:24:06.869336 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:06.869304 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256691b8-991d-4105-9b10-40dbcc6dcc2c" path="/var/lib/kubelet/pods/256691b8-991d-4105-9b10-40dbcc6dcc2c/volumes" Nov 03 14:24:06.869715 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:06.869704 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01491e5-7f57-4419-9186-c2966e8cc22f" path="/var/lib/kubelet/pods/a01491e5-7f57-4419-9186-c2966e8cc22f/volumes" Nov 03 14:24:16.891186 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:24:16.891146 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:24:18.641704 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:18.641666 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz"] Nov 03 14:24:19.560271 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:19.560237 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk"] Nov 03 14:24:29.478037 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:29.478009 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:24:29.488213 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:29.488191 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:24:33.664385 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:33.664341 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" podUID="b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" containerName="main" containerID="cri-o://18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a" gracePeriod=45 Nov 03 14:24:33.930243 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:33.930171 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:24:34.017952 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.017915 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-home\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-home\") pod \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " Nov 03 14:24:34.018175 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018065 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-dshm\") pod \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " Nov 03 14:24:34.018175 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018104 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tls-certs\") pod \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " Nov 03 14:24:34.018175 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018137 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-model-cache\") pod \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " Nov 03 14:24:34.018350 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018187 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-kube-api-access-rprk6\" (UniqueName: \"kubernetes.io/projected/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-kube-api-access-rprk6\") pod \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " Nov 03 14:24:34.018350 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018237 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tmp-dir\") pod \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\" (UID: \"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5\") " Nov 03 14:24:34.018350 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018238 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-home" pod "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" (UID: "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:34.018514 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018472 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-model-cache" pod "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" (UID: "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:34.018593 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018571 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tmp-dir" pod "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" (UID: "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:34.019145 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018946 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:34.019145 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018976 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:34.019145 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.018993 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:34.020655 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.020630 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-kube-api-access-rprk6" pod "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" (UID: "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5"). InnerVolumeSpecName "kube-api-access-rprk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:34.020821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.020800 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-dshm" pod "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" (UID: "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:34.020900 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.020806 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tls-certs" pod "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" (UID: "b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:34.120302 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.120272 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:34.120302 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.120299 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:34.120504 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.120318 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rprk6\" (UniqueName: \"kubernetes.io/projected/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5-kube-api-access-rprk6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:34.243669 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.243583 4541 generic.go:356] "Generic (PLEG): container finished" podID="b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" containerID="18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a" exitCode=0 Nov 03 14:24:34.243669 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.243627 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" Nov 03 14:24:34.243890 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.243665 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" event={"ID":"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5","Type":"ContainerDied","Data":"18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a"} Nov 03 14:24:34.243890 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.243707 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz" event={"ID":"b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5","Type":"ContainerDied","Data":"fd1ea7bce616f4fe8792ddaf56c08e11deefc14892e4e1f7639c230a736e8abb"} Nov 03 14:24:34.243890 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.243726 4541 scope.go:122] "RemoveContainer" containerID="18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a" Nov 03 14:24:34.254273 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.254253 4541 scope.go:122] "RemoveContainer" containerID="18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a" Nov 03 14:24:34.254552 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:24:34.254533 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a\": container with ID starting with 18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a not found: ID does not exist" containerID="18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a" Nov 03 14:24:34.254596 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.254561 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a"} err="failed to get container status \"18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a\": rpc error: code = NotFound desc = could not find container \"18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a\": container with ID starting with 18c071f0df2ad9df2f53c72b205dea79f3e4b5d588c92828e8c84c010b5bc11a not found: ID does not exist" Nov 03 14:24:34.399319 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.399263 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz"] Nov 03 14:24:34.448111 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.448076 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2bceb1/rate-test-model-2bceb1-kserve-6655fc7b8f-zbhjz"] Nov 03 14:24:34.580535 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.580448 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" podUID="ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" containerName="main" containerID="cri-o://d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a" gracePeriod=45 Nov 03 14:24:34.832400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.832342 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:24:34.868292 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.868256 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" path="/var/lib/kubelet/pods/b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5/volumes" Nov 03 14:24:34.928849 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.928818 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tls-certs\") pod \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " Nov 03 14:24:34.929006 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.928859 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tmp-dir\") pod \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " Nov 03 14:24:34.929006 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.928906 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-home\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-home\") pod \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " Nov 03 14:24:34.929123 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.929014 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-kube-api-access-skb55\" (UniqueName: \"kubernetes.io/projected/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-kube-api-access-skb55\") pod \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " Nov 03 14:24:34.929123 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.929083 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-dshm\") pod \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " Nov 03 14:24:34.929222 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.929118 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-model-cache\") pod \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\" (UID: \"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98\") " Nov 03 14:24:34.929222 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.929130 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tmp-dir" pod "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" (UID: "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:34.929317 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.929232 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-home" pod "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" (UID: "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:34.929485 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.929461 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-model-cache" pod "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" (UID: "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:34.929598 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.929578 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:34.929661 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.929605 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:34.929661 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.929619 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:34.931259 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.931231 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-dshm" pod "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" (UID: "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:34.931360 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.931264 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-kube-api-access-skb55" pod "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" (UID: "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98"). InnerVolumeSpecName "kube-api-access-skb55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:34.931360 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:34.931266 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tls-certs" pod "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" (UID: "ed40d4b9-4b0f-4500-ae39-1f6b12d49d98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:35.030794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.030747 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:35.030976 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.030804 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skb55\" (UniqueName: \"kubernetes.io/projected/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-kube-api-access-skb55\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:35.030976 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.030822 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:35.251588 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.251558 4541 generic.go:356] "Generic (PLEG): container finished" podID="ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" containerID="d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a" exitCode=0 Nov 03 14:24:35.251764 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.251617 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" Nov 03 14:24:35.251764 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.251645 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" event={"ID":"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98","Type":"ContainerDied","Data":"d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a"} Nov 03 14:24:35.251764 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.251691 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk" event={"ID":"ed40d4b9-4b0f-4500-ae39-1f6b12d49d98","Type":"ContainerDied","Data":"87476e9863abe19fa368d751472c50ff568dd2fbada3a5a3f99f6d219a4270ab"} Nov 03 14:24:35.251764 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.251710 4541 scope.go:122] "RemoveContainer" containerID="d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a" Nov 03 14:24:35.261497 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.261482 4541 scope.go:122] "RemoveContainer" containerID="d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a" Nov 03 14:24:35.261741 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:24:35.261719 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a\": container with ID starting with d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a not found: ID does not exist" containerID="d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a" Nov 03 14:24:35.261835 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.261751 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a"} err="failed to get container status \"d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a\": rpc error: code = NotFound desc = could not find container \"d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a\": container with ID starting with d3c332f617df8ae23fc6958790ddf6880e9529587fb43fff378b756a381e232a not found: ID does not exist" Nov 03 14:24:35.406985 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.406958 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk"] Nov 03 14:24:35.411766 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:35.411745 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b5daee/test-model-b5daee-kserve-66c67dfc55-qxssk"] Nov 03 14:24:36.868747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:24:36.868702 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" path="/var/lib/kubelet/pods/ed40d4b9-4b0f-4500-ae39-1f6b12d49d98/volumes" Nov 03 14:24:46.889869 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:24:46.889829 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:25:00.775475 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:00.775434 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb"] Nov 03 14:25:07.965746 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:07.965710 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv"] Nov 03 14:25:15.797108 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:15.797060 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" podUID="aafbdc79-6bde-48a0-853c-8a92f80481eb" containerName="main" containerID="cri-o://b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb" gracePeriod=45 Nov 03 14:25:16.068414 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.068390 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 14:25:16.191816 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.191759 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aafbdc79-6bde-48a0-853c-8a92f80481eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/aafbdc79-6bde-48a0-853c-8a92f80481eb-tls-certs\") pod \"aafbdc79-6bde-48a0-853c-8a92f80481eb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " Nov 03 14:25:16.191988 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.191844 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-tmp-dir\") pod \"aafbdc79-6bde-48a0-853c-8a92f80481eb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " Nov 03 14:25:16.191988 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.191872 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-dshm\") pod \"aafbdc79-6bde-48a0-853c-8a92f80481eb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " Nov 03 14:25:16.191988 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.191929 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafbdc79-6bde-48a0-853c-8a92f80481eb-kube-api-access-n8tlj\" (UniqueName: \"kubernetes.io/projected/aafbdc79-6bde-48a0-853c-8a92f80481eb-kube-api-access-n8tlj\") pod \"aafbdc79-6bde-48a0-853c-8a92f80481eb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " Nov 03 14:25:16.192167 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.191997 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-home\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-home\") pod \"aafbdc79-6bde-48a0-853c-8a92f80481eb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " Nov 03 14:25:16.192167 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.192114 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-tmp-dir" pod "aafbdc79-6bde-48a0-853c-8a92f80481eb" (UID: "aafbdc79-6bde-48a0-853c-8a92f80481eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.192167 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.192153 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-model-cache\") pod \"aafbdc79-6bde-48a0-853c-8a92f80481eb\" (UID: \"aafbdc79-6bde-48a0-853c-8a92f80481eb\") " Nov 03 14:25:16.192337 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.192267 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-home" pod "aafbdc79-6bde-48a0-853c-8a92f80481eb" (UID: "aafbdc79-6bde-48a0-853c-8a92f80481eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.192413 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.192368 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-model-cache" pod "aafbdc79-6bde-48a0-853c-8a92f80481eb" (UID: "aafbdc79-6bde-48a0-853c-8a92f80481eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.192641 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.192613 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.192641 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.192631 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.192641 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.192640 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.194148 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.194127 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-dshm" pod "aafbdc79-6bde-48a0-853c-8a92f80481eb" (UID: "aafbdc79-6bde-48a0-853c-8a92f80481eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.194148 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.194136 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aafbdc79-6bde-48a0-853c-8a92f80481eb-tls-certs" pod "aafbdc79-6bde-48a0-853c-8a92f80481eb" (UID: "aafbdc79-6bde-48a0-853c-8a92f80481eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:16.194300 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.194155 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafbdc79-6bde-48a0-853c-8a92f80481eb-kube-api-access-n8tlj" pod "aafbdc79-6bde-48a0-853c-8a92f80481eb" (UID: "aafbdc79-6bde-48a0-853c-8a92f80481eb"). InnerVolumeSpecName "kube-api-access-n8tlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:16.293584 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.293550 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafbdc79-6bde-48a0-853c-8a92f80481eb-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.293584 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.293581 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafbdc79-6bde-48a0-853c-8a92f80481eb-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.293800 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.293592 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8tlj\" (UniqueName: \"kubernetes.io/projected/aafbdc79-6bde-48a0-853c-8a92f80481eb-kube-api-access-n8tlj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.504072 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.504038 4541 generic.go:356] "Generic (PLEG): container finished" podID="aafbdc79-6bde-48a0-853c-8a92f80481eb" containerID="b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb" exitCode=0 Nov 03 14:25:16.504247 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.504087 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" event={"ID":"aafbdc79-6bde-48a0-853c-8a92f80481eb","Type":"ContainerDied","Data":"b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb"} Nov 03 14:25:16.504247 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.504091 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" Nov 03 14:25:16.504247 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.504117 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb" event={"ID":"aafbdc79-6bde-48a0-853c-8a92f80481eb","Type":"ContainerDied","Data":"3c72d089f58719d3fbe185d58ca9f6d9ef5e8678326a21594ab5e6abfd2fa6f3"} Nov 03 14:25:16.504247 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.504135 4541 scope.go:122] "RemoveContainer" containerID="b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb" Nov 03 14:25:16.514444 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.514420 4541 scope.go:122] "RemoveContainer" containerID="b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb" Nov 03 14:25:16.514703 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:25:16.514683 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb\": container with ID starting with b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb not found: ID does not exist" containerID="b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb" Nov 03 14:25:16.514801 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.514710 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb"} err="failed to get container status \"b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb\": rpc error: code = NotFound desc = could not find container \"b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb\": container with ID starting with b49ae8a6cb026ae43cedfec4e50f33a9d5ce830082e4c76f27816b131183bddb not found: ID does not exist" Nov 03 14:25:16.660442 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.660408 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb"] Nov 03 14:25:16.707538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.707506 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/simulator-w0-42c707-kserve-7c4dbd844-s5qqb"] Nov 03 14:25:16.868866 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:16.868786 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafbdc79-6bde-48a0-853c-8a92f80481eb" path="/var/lib/kubelet/pods/aafbdc79-6bde-48a0-853c-8a92f80481eb/volumes" Nov 03 14:25:16.890364 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:25:16.890331 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:25:22.985619 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:22.985566 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" podUID="948aef81-0234-4290-897f-bba6242b630e" containerName="main" containerID="cri-o://14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067" gracePeriod=45 Nov 03 14:25:22.988409 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:22.988380 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" podUID="948aef81-0234-4290-897f-bba6242b630e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.145:8000/ready\": dial tcp 10.133.0.145:8000: connect: connection refused" Nov 03 14:25:23.274919 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.274892 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:25:23.361595 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.361563 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-model-cache\") pod \"948aef81-0234-4290-897f-bba6242b630e\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " Nov 03 14:25:23.361739 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.361642 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-dshm\") pod \"948aef81-0234-4290-897f-bba6242b630e\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " Nov 03 14:25:23.361830 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.361809 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948aef81-0234-4290-897f-bba6242b630e-kube-api-access-ttzdv\" (UniqueName: \"kubernetes.io/projected/948aef81-0234-4290-897f-bba6242b630e-kube-api-access-ttzdv\") pod \"948aef81-0234-4290-897f-bba6242b630e\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " Nov 03 14:25:23.361878 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.361829 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-model-cache" pod "948aef81-0234-4290-897f-bba6242b630e" (UID: "948aef81-0234-4290-897f-bba6242b630e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:23.361985 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.361965 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-tmp-dir\") pod \"948aef81-0234-4290-897f-bba6242b630e\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " Nov 03 14:25:23.362040 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.361998 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948aef81-0234-4290-897f-bba6242b630e-tls-certs\" (UniqueName: \"kubernetes.io/secret/948aef81-0234-4290-897f-bba6242b630e-tls-certs\") pod \"948aef81-0234-4290-897f-bba6242b630e\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " Nov 03 14:25:23.362040 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.362021 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-home\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-home\") pod \"948aef81-0234-4290-897f-bba6242b630e\" (UID: \"948aef81-0234-4290-897f-bba6242b630e\") " Nov 03 14:25:23.362242 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.362211 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-tmp-dir" pod "948aef81-0234-4290-897f-bba6242b630e" (UID: "948aef81-0234-4290-897f-bba6242b630e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:23.362373 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.362346 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-home" pod "948aef81-0234-4290-897f-bba6242b630e" (UID: "948aef81-0234-4290-897f-bba6242b630e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:23.362437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.362406 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:23.362437 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.362428 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:23.363910 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.363881 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-dshm" pod "948aef81-0234-4290-897f-bba6242b630e" (UID: "948aef81-0234-4290-897f-bba6242b630e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:23.364006 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.363956 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948aef81-0234-4290-897f-bba6242b630e-kube-api-access-ttzdv" pod "948aef81-0234-4290-897f-bba6242b630e" (UID: "948aef81-0234-4290-897f-bba6242b630e"). InnerVolumeSpecName "kube-api-access-ttzdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:23.364129 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.364108 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948aef81-0234-4290-897f-bba6242b630e-tls-certs" pod "948aef81-0234-4290-897f-bba6242b630e" (UID: "948aef81-0234-4290-897f-bba6242b630e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:23.463910 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.463883 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttzdv\" (UniqueName: \"kubernetes.io/projected/948aef81-0234-4290-897f-bba6242b630e-kube-api-access-ttzdv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:23.463910 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.463909 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/948aef81-0234-4290-897f-bba6242b630e-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:23.464083 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.463919 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:23.464083 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.463927 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/948aef81-0234-4290-897f-bba6242b630e-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:23.543195 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.543121 4541 generic.go:356] "Generic (PLEG): container finished" podID="948aef81-0234-4290-897f-bba6242b630e" containerID="14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067" exitCode=0 Nov 03 14:25:23.543195 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.543165 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" Nov 03 14:25:23.543366 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.543214 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" event={"ID":"948aef81-0234-4290-897f-bba6242b630e","Type":"ContainerDied","Data":"14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067"} Nov 03 14:25:23.543366 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.543250 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv" event={"ID":"948aef81-0234-4290-897f-bba6242b630e","Type":"ContainerDied","Data":"bbe04203bf5c2d9c33e3221179fe8edf339a36b24ca209679435d30228a8be1f"} Nov 03 14:25:23.543366 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.543271 4541 scope.go:122] "RemoveContainer" containerID="14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067" Nov 03 14:25:23.552752 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.552725 4541 scope.go:122] "RemoveContainer" containerID="14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067" Nov 03 14:25:23.553060 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:25:23.553039 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067\": container with ID starting with 14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067 not found: ID does not exist" containerID="14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067" Nov 03 14:25:23.553111 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.553068 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067"} err="failed to get container status \"14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067\": rpc error: code = NotFound desc = could not find container \"14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067\": container with ID starting with 14de2af6dc3b9750d8ac37d821f8e80addb26e9ab4776141b6c1391167c04067 not found: ID does not exist" Nov 03 14:25:23.699080 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.699054 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv"] Nov 03 14:25:23.705896 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:23.705871 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-d341af/rate-test-model-d341af-kserve-546c59794f-5x5hv"] Nov 03 14:25:24.868158 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:24.868124 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948aef81-0234-4290-897f-bba6242b630e" path="/var/lib/kubelet/pods/948aef81-0234-4290-897f-bba6242b630e/volumes" Nov 03 14:25:46.620365 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:46.620326 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn"] Nov 03 14:25:46.620876 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:46.620527 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" podUID="2798c769-768f-4eff-9944-61f6dc97e873" containerName="maas-api" containerID="cri-o://2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6" gracePeriod=30 Nov 03 14:25:46.890440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:46.890414 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:25:46.892348 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:25:46.892316 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:25:47.035596 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.035555 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2798c769-768f-4eff-9944-61f6dc97e873-kube-api-access-2mzx2\" (UniqueName: \"kubernetes.io/projected/2798c769-768f-4eff-9944-61f6dc97e873-kube-api-access-2mzx2\") pod \"2798c769-768f-4eff-9944-61f6dc97e873\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " Nov 03 14:25:47.035816 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.035692 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2798c769-768f-4eff-9944-61f6dc97e873-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2798c769-768f-4eff-9944-61f6dc97e873-openshift-service-ca\") pod \"2798c769-768f-4eff-9944-61f6dc97e873\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " Nov 03 14:25:47.035910 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.035813 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-metrics-certs\") pod \"2798c769-768f-4eff-9944-61f6dc97e873\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " Nov 03 14:25:47.035974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.035906 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-maas-api-tls\") pod \"2798c769-768f-4eff-9944-61f6dc97e873\" (UID: \"2798c769-768f-4eff-9944-61f6dc97e873\") " Nov 03 14:25:47.036154 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.036122 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2798c769-768f-4eff-9944-61f6dc97e873-openshift-service-ca" pod "2798c769-768f-4eff-9944-61f6dc97e873" (UID: "2798c769-768f-4eff-9944-61f6dc97e873"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:47.036408 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.036376 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2798c769-768f-4eff-9944-61f6dc97e873-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:47.038425 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.038387 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2798c769-768f-4eff-9944-61f6dc97e873-kube-api-access-2mzx2" pod "2798c769-768f-4eff-9944-61f6dc97e873" (UID: "2798c769-768f-4eff-9944-61f6dc97e873"). InnerVolumeSpecName "kube-api-access-2mzx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:47.038534 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.038434 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-maas-api-tls" pod "2798c769-768f-4eff-9944-61f6dc97e873" (UID: "2798c769-768f-4eff-9944-61f6dc97e873"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:47.038534 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.038478 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-metrics-certs" pod "2798c769-768f-4eff-9944-61f6dc97e873" (UID: "2798c769-768f-4eff-9944-61f6dc97e873"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:47.137562 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.137481 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:47.137562 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.137507 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2798c769-768f-4eff-9944-61f6dc97e873-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:47.137562 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.137517 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mzx2\" (UniqueName: \"kubernetes.io/projected/2798c769-768f-4eff-9944-61f6dc97e873-kube-api-access-2mzx2\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:47.675874 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.675841 4541 generic.go:356] "Generic (PLEG): container finished" podID="2798c769-768f-4eff-9944-61f6dc97e873" containerID="2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6" exitCode=0 Nov 03 14:25:47.676288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.675889 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" Nov 03 14:25:47.676288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.675932 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" event={"ID":"2798c769-768f-4eff-9944-61f6dc97e873","Type":"ContainerDied","Data":"2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6"} Nov 03 14:25:47.676288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.675982 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn" event={"ID":"2798c769-768f-4eff-9944-61f6dc97e873","Type":"ContainerDied","Data":"d46664b66935b1932edb3fe8b2e6e22de45e6f8bb7112e0132c9922e64555456"} Nov 03 14:25:47.676288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.676004 4541 scope.go:122] "RemoveContainer" containerID="2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6" Nov 03 14:25:47.686337 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.686317 4541 scope.go:122] "RemoveContainer" containerID="2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6" Nov 03 14:25:47.686598 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:25:47.686578 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6\": container with ID starting with 2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6 not found: ID does not exist" containerID="2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6" Nov 03 14:25:47.686647 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.686607 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6"} err="failed to get container status \"2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6\": rpc error: code = NotFound desc = could not find container \"2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6\": container with ID starting with 2cbe1d3f22b5fb32f3168ca34b03b3e7b2c5a0873ec75dadf110649eff426ac6 not found: ID does not exist" Nov 03 14:25:47.831517 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.831485 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn"] Nov 03 14:25:47.878972 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:47.878931 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b5daee-6d5d98bbcd-8lmpn"] Nov 03 14:25:48.870625 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:48.870590 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2798c769-768f-4eff-9944-61f6dc97e873" path="/var/lib/kubelet/pods/2798c769-768f-4eff-9944-61f6dc97e873/volumes" Nov 03 14:25:58.410865 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.410828 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq"] Nov 03 14:25:58.411355 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.411035 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" podUID="cd08615b-547e-4e1d-a572-70f216d10e41" containerName="maas-api" containerID="cri-o://8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f" gracePeriod=30 Nov 03 14:25:58.680655 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.680582 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 14:25:58.745967 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.745935 4541 generic.go:356] "Generic (PLEG): container finished" podID="cd08615b-547e-4e1d-a572-70f216d10e41" containerID="8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f" exitCode=0 Nov 03 14:25:58.746148 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.745980 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" Nov 03 14:25:58.746148 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.746004 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" event={"ID":"cd08615b-547e-4e1d-a572-70f216d10e41","Type":"ContainerDied","Data":"8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f"} Nov 03 14:25:58.746148 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.746048 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq" event={"ID":"cd08615b-547e-4e1d-a572-70f216d10e41","Type":"ContainerDied","Data":"33fa8cd980bf97570cadc0d1b25f35707dc28acddf2b4e5eebf597d09be1fdab"} Nov 03 14:25:58.746148 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.746069 4541 scope.go:122] "RemoveContainer" containerID="8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f" Nov 03 14:25:58.755036 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.755017 4541 scope.go:122] "RemoveContainer" containerID="8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f" Nov 03 14:25:58.755331 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:25:58.755310 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f\": container with ID starting with 8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f not found: ID does not exist" containerID="8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f" Nov 03 14:25:58.755382 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.755340 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f"} err="failed to get container status \"8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f\": rpc error: code = NotFound desc = could not find container \"8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f\": container with ID starting with 8ab3a65670c33fcf56a1832fb9a9e8dcf385876499357c23ae5935c4f80ff26f not found: ID does not exist" Nov 03 14:25:58.758669 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.758647 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd08615b-547e-4e1d-a572-70f216d10e41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd08615b-547e-4e1d-a572-70f216d10e41-openshift-service-ca\") pod \"cd08615b-547e-4e1d-a572-70f216d10e41\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " Nov 03 14:25:58.758870 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.758854 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-metrics-certs\") pod \"cd08615b-547e-4e1d-a572-70f216d10e41\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " Nov 03 14:25:58.758949 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.758899 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls\") pod \"cd08615b-547e-4e1d-a572-70f216d10e41\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " Nov 03 14:25:58.758949 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.758936 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd08615b-547e-4e1d-a572-70f216d10e41-kube-api-access-r74ll\" (UniqueName: \"kubernetes.io/projected/cd08615b-547e-4e1d-a572-70f216d10e41-kube-api-access-r74ll\") pod \"cd08615b-547e-4e1d-a572-70f216d10e41\" (UID: \"cd08615b-547e-4e1d-a572-70f216d10e41\") " Nov 03 14:25:58.758949 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.758939 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd08615b-547e-4e1d-a572-70f216d10e41-openshift-service-ca" pod "cd08615b-547e-4e1d-a572-70f216d10e41" (UID: "cd08615b-547e-4e1d-a572-70f216d10e41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:58.759368 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.759345 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd08615b-547e-4e1d-a572-70f216d10e41-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:58.760969 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.760946 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls" pod "cd08615b-547e-4e1d-a572-70f216d10e41" (UID: "cd08615b-547e-4e1d-a572-70f216d10e41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:58.761063 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.760990 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd08615b-547e-4e1d-a572-70f216d10e41-kube-api-access-r74ll" pod "cd08615b-547e-4e1d-a572-70f216d10e41" (UID: "cd08615b-547e-4e1d-a572-70f216d10e41"). InnerVolumeSpecName "kube-api-access-r74ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:58.761063 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.761004 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-metrics-certs" pod "cd08615b-547e-4e1d-a572-70f216d10e41" (UID: "cd08615b-547e-4e1d-a572-70f216d10e41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:58.860094 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.860062 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:58.860094 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.860088 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd08615b-547e-4e1d-a572-70f216d10e41-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:58.860094 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:58.860099 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r74ll\" (UniqueName: \"kubernetes.io/projected/cd08615b-547e-4e1d-a572-70f216d10e41-kube-api-access-r74ll\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:59.156747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:59.156712 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq"] Nov 03 14:25:59.160518 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:25:59.160493 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-42c707-7b4c95bf45-rrgpq"] Nov 03 14:26:00.867241 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:26:00.867205 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd08615b-547e-4e1d-a572-70f216d10e41" path="/var/lib/kubelet/pods/cd08615b-547e-4e1d-a572-70f216d10e41/volumes" Nov 03 14:26:16.888797 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:26:16.888709 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:26:46.890721 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:26:46.890656 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:27:16.889593 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:27:16.889552 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:27:46.890467 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:27:46.890428 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:27:48.871579 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:27:48.871547 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx"] Nov 03 14:27:48.970897 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:27:48.970860 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8"] Nov 03 14:27:53.872797 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:27:53.872728 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" podUID="16e8525a-86b2-440e-8215-3431a27a6a54" containerName="payload-processing" containerID="cri-o://776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638" gracePeriod=25 Nov 03 14:27:53.971672 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:27:53.971626 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" podUID="9890570b-67ea-47e3-b607-c1456210c7a6" containerName="payload-pre-processing" containerID="cri-o://59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10" gracePeriod=25 Nov 03 14:27:58.643487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:27:58.643448 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" podUID="9890570b-67ea-47e3-b607-c1456210c7a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:27:58.643949 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:27:58.643454 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" podUID="16e8525a-86b2-440e-8215-3431a27a6a54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:08.658794 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:08.654306 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" podUID="9890570b-67ea-47e3-b607-c1456210c7a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:08.664792 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:08.660867 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" podUID="16e8525a-86b2-440e-8215-3431a27a6a54" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:14.131792 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.131705 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 14:28:14.231111 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.231082 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 14:28:14.248415 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.248379 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e8525a-86b2-440e-8215-3431a27a6a54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e8525a-86b2-440e-8215-3431a27a6a54-plugins-config-volume\") pod \"16e8525a-86b2-440e-8215-3431a27a6a54\" (UID: \"16e8525a-86b2-440e-8215-3431a27a6a54\") " Nov 03 14:28:14.248585 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.248500 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e8525a-86b2-440e-8215-3431a27a6a54-kube-api-access-w56rx\" (UniqueName: \"kubernetes.io/projected/16e8525a-86b2-440e-8215-3431a27a6a54-kube-api-access-w56rx\") pod \"16e8525a-86b2-440e-8215-3431a27a6a54\" (UID: \"16e8525a-86b2-440e-8215-3431a27a6a54\") " Nov 03 14:28:14.248866 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.248834 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e8525a-86b2-440e-8215-3431a27a6a54-plugins-config-volume" pod "16e8525a-86b2-440e-8215-3431a27a6a54" (UID: "16e8525a-86b2-440e-8215-3431a27a6a54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:14.248966 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.248948 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e8525a-86b2-440e-8215-3431a27a6a54-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:14.250904 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.250880 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e8525a-86b2-440e-8215-3431a27a6a54-kube-api-access-w56rx" pod "16e8525a-86b2-440e-8215-3431a27a6a54" (UID: "16e8525a-86b2-440e-8215-3431a27a6a54"). InnerVolumeSpecName "kube-api-access-w56rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:14.349338 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.349301 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9890570b-67ea-47e3-b607-c1456210c7a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9890570b-67ea-47e3-b607-c1456210c7a6-plugins-config-volume\") pod \"9890570b-67ea-47e3-b607-c1456210c7a6\" (UID: \"9890570b-67ea-47e3-b607-c1456210c7a6\") " Nov 03 14:28:14.349573 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.349463 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9890570b-67ea-47e3-b607-c1456210c7a6-kube-api-access-z9n6w\" (UniqueName: \"kubernetes.io/projected/9890570b-67ea-47e3-b607-c1456210c7a6-kube-api-access-z9n6w\") pod \"9890570b-67ea-47e3-b607-c1456210c7a6\" (UID: \"9890570b-67ea-47e3-b607-c1456210c7a6\") " Nov 03 14:28:14.349824 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.349736 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9890570b-67ea-47e3-b607-c1456210c7a6-plugins-config-volume" pod "9890570b-67ea-47e3-b607-c1456210c7a6" (UID: "9890570b-67ea-47e3-b607-c1456210c7a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:14.350017 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.349998 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56rx\" (UniqueName: \"kubernetes.io/projected/16e8525a-86b2-440e-8215-3431a27a6a54-kube-api-access-w56rx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:14.350089 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.350021 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9890570b-67ea-47e3-b607-c1456210c7a6-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:14.352240 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.352212 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9890570b-67ea-47e3-b607-c1456210c7a6-kube-api-access-z9n6w" pod "9890570b-67ea-47e3-b607-c1456210c7a6" (UID: "9890570b-67ea-47e3-b607-c1456210c7a6"). InnerVolumeSpecName "kube-api-access-z9n6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:14.450757 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.450720 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9n6w\" (UniqueName: \"kubernetes.io/projected/9890570b-67ea-47e3-b607-c1456210c7a6-kube-api-access-z9n6w\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:14.505989 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.505951 4541 generic.go:356] "Generic (PLEG): container finished" podID="16e8525a-86b2-440e-8215-3431a27a6a54" containerID="776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638" exitCode=0 Nov 03 14:28:14.506178 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.506003 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" Nov 03 14:28:14.506178 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.506035 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" event={"ID":"16e8525a-86b2-440e-8215-3431a27a6a54","Type":"ContainerDied","Data":"776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638"} Nov 03 14:28:14.506178 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.506092 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx" event={"ID":"16e8525a-86b2-440e-8215-3431a27a6a54","Type":"ContainerDied","Data":"6febd9cf3d6839f80df71ee54687b9541332b80fc43b5bc030195a2ab17eb40c"} Nov 03 14:28:14.506178 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.506125 4541 scope.go:122] "RemoveContainer" containerID="776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638" Nov 03 14:28:14.507469 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.507444 4541 generic.go:356] "Generic (PLEG): container finished" podID="9890570b-67ea-47e3-b607-c1456210c7a6" containerID="59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10" exitCode=0 Nov 03 14:28:14.507594 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.507486 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" Nov 03 14:28:14.507594 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.507520 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" event={"ID":"9890570b-67ea-47e3-b607-c1456210c7a6","Type":"ContainerDied","Data":"59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10"} Nov 03 14:28:14.507594 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.507572 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8" event={"ID":"9890570b-67ea-47e3-b607-c1456210c7a6","Type":"ContainerDied","Data":"35d0a01b489ce82a3eb6bc3266b6df2ee1221c9527dd916133ee75559d14f596"} Nov 03 14:28:14.517353 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.517335 4541 scope.go:122] "RemoveContainer" containerID="776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638" Nov 03 14:28:14.517656 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:28:14.517632 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638\": container with ID starting with 776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638 not found: ID does not exist" containerID="776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638" Nov 03 14:28:14.517736 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.517665 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638"} err="failed to get container status \"776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638\": rpc error: code = NotFound desc = could not find container \"776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638\": container with ID starting with 776ee52fc572df138425dd8847835737330cba8fce34f85d7e2e731aa5c80638 not found: ID does not exist" Nov 03 14:28:14.517736 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.517687 4541 scope.go:122] "RemoveContainer" containerID="59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10" Nov 03 14:28:14.526706 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.526684 4541 scope.go:122] "RemoveContainer" containerID="59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10" Nov 03 14:28:14.527004 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:28:14.526984 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10\": container with ID starting with 59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10 not found: ID does not exist" containerID="59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10" Nov 03 14:28:14.527121 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.527015 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10"} err="failed to get container status \"59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10\": rpc error: code = NotFound desc = could not find container \"59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10\": container with ID starting with 59f59de7486faaa68d42cc066b0f7db979172638c99e51f84a3a3fa28c486e10 not found: ID does not exist" Nov 03 14:28:14.662675 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.662639 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8"] Nov 03 14:28:14.666483 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.666453 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-5ae63d-65f9d6f987-4mcj8"] Nov 03 14:28:14.770220 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.770132 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx"] Nov 03 14:28:14.774556 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.774526 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-5ae63d-58578cc7b9-wx6dx"] Nov 03 14:28:14.868457 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.868416 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e8525a-86b2-440e-8215-3431a27a6a54" path="/var/lib/kubelet/pods/16e8525a-86b2-440e-8215-3431a27a6a54/volumes" Nov 03 14:28:14.868847 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:14.868832 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9890570b-67ea-47e3-b607-c1456210c7a6" path="/var/lib/kubelet/pods/9890570b-67ea-47e3-b607-c1456210c7a6/volumes" Nov 03 14:28:16.890331 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:28:16.890293 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:28:30.635384 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:30.635344 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh"] Nov 03 14:28:30.635933 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:30.635598 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" containerID="cri-o://e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680" gracePeriod=30 Nov 03 14:28:34.730067 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:34.730025 4541 patch_prober.go:29] interesting pod/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:28:34.730461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:34.730123 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:28:35.892174 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.892150 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 14:28:35.984337 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984286 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-envoy\") pod \"e185d31d-49f8-4a83-a724-5b2219356e3d\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " Nov 03 14:28:35.984525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984369 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-token\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-token\") pod \"e185d31d-49f8-4a83-a724-5b2219356e3d\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " Nov 03 14:28:35.984525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984399 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-credential-socket\") pod \"e185d31d-49f8-4a83-a724-5b2219356e3d\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " Nov 03 14:28:35.984525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984446 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-podinfo\") pod \"e185d31d-49f8-4a83-a724-5b2219356e3d\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " Nov 03 14:28:35.984525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984473 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-socket\") pod \"e185d31d-49f8-4a83-a724-5b2219356e3d\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " Nov 03 14:28:35.984525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984520 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e185d31d-49f8-4a83-a724-5b2219356e3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185d31d-49f8-4a83-a724-5b2219356e3d-istiod-ca-cert\") pod \"e185d31d-49f8-4a83-a724-5b2219356e3d\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " Nov 03 14:28:35.984832 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984544 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-kube-api-access-npdhc\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-kube-api-access-npdhc\") pod \"e185d31d-49f8-4a83-a724-5b2219356e3d\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " Nov 03 14:28:35.984832 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984586 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-certs\") pod \"e185d31d-49f8-4a83-a724-5b2219356e3d\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " Nov 03 14:28:35.984832 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984628 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-data\") pod \"e185d31d-49f8-4a83-a724-5b2219356e3d\" (UID: \"e185d31d-49f8-4a83-a724-5b2219356e3d\") " Nov 03 14:28:35.984832 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984715 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-credential-socket" pod "e185d31d-49f8-4a83-a724-5b2219356e3d" (UID: "e185d31d-49f8-4a83-a724-5b2219356e3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:35.985060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984914 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-socket" pod "e185d31d-49f8-4a83-a724-5b2219356e3d" (UID: "e185d31d-49f8-4a83-a724-5b2219356e3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:35.985060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984949 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-data" pod "e185d31d-49f8-4a83-a724-5b2219356e3d" (UID: "e185d31d-49f8-4a83-a724-5b2219356e3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:35.985060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.984973 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-certs" pod "e185d31d-49f8-4a83-a724-5b2219356e3d" (UID: "e185d31d-49f8-4a83-a724-5b2219356e3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:35.985060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.985007 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e185d31d-49f8-4a83-a724-5b2219356e3d-istiod-ca-cert" pod "e185d31d-49f8-4a83-a724-5b2219356e3d" (UID: "e185d31d-49f8-4a83-a724-5b2219356e3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:35.985267 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.985181 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:35.985267 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.985203 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:35.985267 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.985218 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:35.985267 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.985232 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:35.985267 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.985247 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e185d31d-49f8-4a83-a724-5b2219356e3d-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:35.986600 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.986560 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-envoy" pod "e185d31d-49f8-4a83-a724-5b2219356e3d" (UID: "e185d31d-49f8-4a83-a724-5b2219356e3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:35.986723 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.986609 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-token" pod "e185d31d-49f8-4a83-a724-5b2219356e3d" (UID: "e185d31d-49f8-4a83-a724-5b2219356e3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:35.987041 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.987024 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-podinfo" pod "e185d31d-49f8-4a83-a724-5b2219356e3d" (UID: "e185d31d-49f8-4a83-a724-5b2219356e3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:28:35.987185 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:35.987167 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-kube-api-access-npdhc" pod "e185d31d-49f8-4a83-a724-5b2219356e3d" (UID: "e185d31d-49f8-4a83-a724-5b2219356e3d"). InnerVolumeSpecName "kube-api-access-npdhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:36.086313 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.086280 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:36.086313 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.086310 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:36.086517 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.086323 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e185d31d-49f8-4a83-a724-5b2219356e3d-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:36.086517 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.086337 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npdhc\" (UniqueName: \"kubernetes.io/projected/e185d31d-49f8-4a83-a724-5b2219356e3d-kube-api-access-npdhc\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:36.639553 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.639520 4541 generic.go:356] "Generic (PLEG): container finished" podID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerID="e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680" exitCode=0 Nov 03 14:28:36.639735 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.639572 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" Nov 03 14:28:36.639735 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.639601 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" event={"ID":"e185d31d-49f8-4a83-a724-5b2219356e3d","Type":"ContainerDied","Data":"e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680"} Nov 03 14:28:36.639735 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.639663 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh" event={"ID":"e185d31d-49f8-4a83-a724-5b2219356e3d","Type":"ContainerDied","Data":"8f2b89794d506975452e0f3e89de09a9de2c0e2703bd7df245ba00d9d35b8c60"} Nov 03 14:28:36.639735 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.639692 4541 scope.go:122] "RemoveContainer" containerID="e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680" Nov 03 14:28:36.649926 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.649906 4541 scope.go:122] "RemoveContainer" containerID="1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e" Nov 03 14:28:36.658751 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.658733 4541 scope.go:122] "RemoveContainer" containerID="e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680" Nov 03 14:28:36.659072 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:28:36.659053 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680\": container with ID starting with e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680 not found: ID does not exist" containerID="e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680" Nov 03 14:28:36.659158 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.659088 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680"} err="failed to get container status \"e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680\": rpc error: code = NotFound desc = could not find container \"e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680\": container with ID starting with e4bda7e94f7cb145fc86d279e33ce43e9714298eb9094da15514cde94ce79680 not found: ID does not exist" Nov 03 14:28:36.659158 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.659117 4541 scope.go:122] "RemoveContainer" containerID="1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e" Nov 03 14:28:36.659363 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:28:36.659341 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e\": container with ID starting with 1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e not found: ID does not exist" containerID="1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e" Nov 03 14:28:36.659403 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.659371 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e"} err="failed to get container status \"1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e\": rpc error: code = NotFound desc = could not find container \"1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e\": container with ID starting with 1c7cb102a013628f292385b34d444ae71f2d0491d29c0dd51f2a67b853de2c8e not found: ID does not exist" Nov 03 14:28:36.795950 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.795914 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh"] Nov 03 14:28:36.801689 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.801663 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-5ae63d-openshift-default-5dc88f8575-pfkxh"] Nov 03 14:28:36.868920 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:28:36.868886 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" path="/var/lib/kubelet/pods/e185d31d-49f8-4a83-a724-5b2219356e3d/volumes" Nov 03 14:28:46.891658 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:28:46.891611 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:29:09.978535 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:09.978448 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl"] Nov 03 14:29:10.081493 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:10.081457 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt"] Nov 03 14:29:11.827799 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:11.827740 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv"] Nov 03 14:29:11.828246 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:11.827945 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" podUID="4f83df78-c34c-4c09-b36a-f819e54ddebb" containerName="maas-api" containerID="cri-o://a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa" gracePeriod=30 Nov 03 14:29:12.081414 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.081355 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 14:29:12.188520 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.188481 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-metrics-certs\") pod \"4f83df78-c34c-4c09-b36a-f819e54ddebb\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " Nov 03 14:29:12.188719 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.188552 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f83df78-c34c-4c09-b36a-f819e54ddebb-kube-api-access-tc8b7\" (UniqueName: \"kubernetes.io/projected/4f83df78-c34c-4c09-b36a-f819e54ddebb-kube-api-access-tc8b7\") pod \"4f83df78-c34c-4c09-b36a-f819e54ddebb\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " Nov 03 14:29:12.188719 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.188613 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f83df78-c34c-4c09-b36a-f819e54ddebb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83df78-c34c-4c09-b36a-f819e54ddebb-openshift-service-ca\") pod \"4f83df78-c34c-4c09-b36a-f819e54ddebb\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " Nov 03 14:29:12.188719 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.188710 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls\") pod \"4f83df78-c34c-4c09-b36a-f819e54ddebb\" (UID: \"4f83df78-c34c-4c09-b36a-f819e54ddebb\") " Nov 03 14:29:12.189060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.189029 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f83df78-c34c-4c09-b36a-f819e54ddebb-openshift-service-ca" pod "4f83df78-c34c-4c09-b36a-f819e54ddebb" (UID: "4f83df78-c34c-4c09-b36a-f819e54ddebb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:12.189210 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.189191 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f83df78-c34c-4c09-b36a-f819e54ddebb-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:12.190700 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.190679 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-metrics-certs" pod "4f83df78-c34c-4c09-b36a-f819e54ddebb" (UID: "4f83df78-c34c-4c09-b36a-f819e54ddebb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:12.191170 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.191139 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls" pod "4f83df78-c34c-4c09-b36a-f819e54ddebb" (UID: "4f83df78-c34c-4c09-b36a-f819e54ddebb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:12.191284 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.191186 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f83df78-c34c-4c09-b36a-f819e54ddebb-kube-api-access-tc8b7" pod "4f83df78-c34c-4c09-b36a-f819e54ddebb" (UID: "4f83df78-c34c-4c09-b36a-f819e54ddebb"). InnerVolumeSpecName "kube-api-access-tc8b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:12.289980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.289946 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc8b7\" (UniqueName: \"kubernetes.io/projected/4f83df78-c34c-4c09-b36a-f819e54ddebb-kube-api-access-tc8b7\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:12.289980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.289977 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:12.289980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.289987 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f83df78-c34c-4c09-b36a-f819e54ddebb-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:12.832099 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.832062 4541 generic.go:356] "Generic (PLEG): container finished" podID="4f83df78-c34c-4c09-b36a-f819e54ddebb" containerID="a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa" exitCode=0 Nov 03 14:29:12.832543 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.832108 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" Nov 03 14:29:12.832543 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.832119 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" event={"ID":"4f83df78-c34c-4c09-b36a-f819e54ddebb","Type":"ContainerDied","Data":"a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa"} Nov 03 14:29:12.832543 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.832163 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv" event={"ID":"4f83df78-c34c-4c09-b36a-f819e54ddebb","Type":"ContainerDied","Data":"10486ba109a4b6d9c3cb3b11927057cc8279dd06e917d340a8df9b8340ff5d01"} Nov 03 14:29:12.832543 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.832187 4541 scope.go:122] "RemoveContainer" containerID="a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa" Nov 03 14:29:12.845240 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.845215 4541 scope.go:122] "RemoveContainer" containerID="a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa" Nov 03 14:29:12.845575 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:29:12.845554 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa\": container with ID starting with a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa not found: ID does not exist" containerID="a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa" Nov 03 14:29:12.845637 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.845586 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa"} err="failed to get container status \"a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa\": rpc error: code = NotFound desc = could not find container \"a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa\": container with ID starting with a1699233f594f898d0f3587a7e0165c457656761f4d881667f2430a96cbc0caa not found: ID does not exist" Nov 03 14:29:12.987693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:12.987658 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv"] Nov 03 14:29:13.034890 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:13.034847 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-efbdec-5d47446449-xzhpv"] Nov 03 14:29:14.868480 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:14.868447 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f83df78-c34c-4c09-b36a-f819e54ddebb" path="/var/lib/kubelet/pods/4f83df78-c34c-4c09-b36a-f819e54ddebb/volumes" Nov 03 14:29:14.979610 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:14.979570 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" podUID="1f4cf945-e803-4428-9287-633fb3653273" containerName="payload-processing" containerID="cri-o://f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b" gracePeriod=25 Nov 03 14:29:15.082255 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:15.082189 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" podUID="fdff9c10-2c0e-4384-a569-099f2d0800ba" containerName="payload-pre-processing" containerID="cri-o://40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43" gracePeriod=25 Nov 03 14:29:16.890245 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:29:16.890197 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:29:22.992575 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:22.992535 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" podUID="fdff9c10-2c0e-4384-a569-099f2d0800ba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:29:22.992986 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:22.992597 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" podUID="1f4cf945-e803-4428-9287-633fb3653273" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:29:23.759851 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:23.759814 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt"] Nov 03 14:29:23.806547 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:23.806513 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn"] Nov 03 14:29:29.509878 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:29.509846 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:29:29.520297 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:29.520263 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:29:32.573482 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:32.573445 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p"] Nov 03 14:29:32.690468 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:32.690345 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh"] Nov 03 14:29:32.992015 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:32.991974 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" podUID="fdff9c10-2c0e-4384-a569-099f2d0800ba" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:29:32.992015 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:32.992005 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" podUID="1f4cf945-e803-4428-9287-633fb3653273" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:29:35.250223 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.250162 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 14:29:35.336586 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.336559 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4cf945-e803-4428-9287-633fb3653273-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4cf945-e803-4428-9287-633fb3653273-plugins-config-volume\") pod \"1f4cf945-e803-4428-9287-633fb3653273\" (UID: \"1f4cf945-e803-4428-9287-633fb3653273\") " Nov 03 14:29:35.336726 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.336640 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4cf945-e803-4428-9287-633fb3653273-kube-api-access-x8lg6\" (UniqueName: \"kubernetes.io/projected/1f4cf945-e803-4428-9287-633fb3653273-kube-api-access-x8lg6\") pod \"1f4cf945-e803-4428-9287-633fb3653273\" (UID: \"1f4cf945-e803-4428-9287-633fb3653273\") " Nov 03 14:29:35.336994 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.336967 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4cf945-e803-4428-9287-633fb3653273-plugins-config-volume" pod "1f4cf945-e803-4428-9287-633fb3653273" (UID: "1f4cf945-e803-4428-9287-633fb3653273"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:35.337108 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.337083 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4cf945-e803-4428-9287-633fb3653273-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:35.338846 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.338821 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4cf945-e803-4428-9287-633fb3653273-kube-api-access-x8lg6" pod "1f4cf945-e803-4428-9287-633fb3653273" (UID: "1f4cf945-e803-4428-9287-633fb3653273"). InnerVolumeSpecName "kube-api-access-x8lg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:35.341695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.341675 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 14:29:35.438332 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.438302 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdff9c10-2c0e-4384-a569-099f2d0800ba-kube-api-access-2mbxf\" (UniqueName: \"kubernetes.io/projected/fdff9c10-2c0e-4384-a569-099f2d0800ba-kube-api-access-2mbxf\") pod \"fdff9c10-2c0e-4384-a569-099f2d0800ba\" (UID: \"fdff9c10-2c0e-4384-a569-099f2d0800ba\") " Nov 03 14:29:35.438515 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.438341 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdff9c10-2c0e-4384-a569-099f2d0800ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdff9c10-2c0e-4384-a569-099f2d0800ba-plugins-config-volume\") pod \"fdff9c10-2c0e-4384-a569-099f2d0800ba\" (UID: \"fdff9c10-2c0e-4384-a569-099f2d0800ba\") " Nov 03 14:29:35.438699 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.438674 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8lg6\" (UniqueName: \"kubernetes.io/projected/1f4cf945-e803-4428-9287-633fb3653273-kube-api-access-x8lg6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:35.438838 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.438796 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdff9c10-2c0e-4384-a569-099f2d0800ba-plugins-config-volume" pod "fdff9c10-2c0e-4384-a569-099f2d0800ba" (UID: "fdff9c10-2c0e-4384-a569-099f2d0800ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:35.440394 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.440370 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdff9c10-2c0e-4384-a569-099f2d0800ba-kube-api-access-2mbxf" pod "fdff9c10-2c0e-4384-a569-099f2d0800ba" (UID: "fdff9c10-2c0e-4384-a569-099f2d0800ba"). InnerVolumeSpecName "kube-api-access-2mbxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:35.539715 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.539632 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mbxf\" (UniqueName: \"kubernetes.io/projected/fdff9c10-2c0e-4384-a569-099f2d0800ba-kube-api-access-2mbxf\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:35.539715 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.539662 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdff9c10-2c0e-4384-a569-099f2d0800ba-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:35.954604 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.954571 4541 generic.go:356] "Generic (PLEG): container finished" podID="fdff9c10-2c0e-4384-a569-099f2d0800ba" containerID="40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43" exitCode=0 Nov 03 14:29:35.954818 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.954619 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" Nov 03 14:29:35.954818 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.954643 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" event={"ID":"fdff9c10-2c0e-4384-a569-099f2d0800ba","Type":"ContainerDied","Data":"40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43"} Nov 03 14:29:35.954818 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.954701 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt" event={"ID":"fdff9c10-2c0e-4384-a569-099f2d0800ba","Type":"ContainerDied","Data":"72a16093df3309f9b8eb72576b44882b9efcb1bad57b3784aab222caee0bb51b"} Nov 03 14:29:35.954818 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.954735 4541 scope.go:122] "RemoveContainer" containerID="40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43" Nov 03 14:29:35.956073 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.956052 4541 generic.go:356] "Generic (PLEG): container finished" podID="1f4cf945-e803-4428-9287-633fb3653273" containerID="f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b" exitCode=0 Nov 03 14:29:35.956156 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.956098 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" Nov 03 14:29:35.956156 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.956124 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" event={"ID":"1f4cf945-e803-4428-9287-633fb3653273","Type":"ContainerDied","Data":"f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b"} Nov 03 14:29:35.956260 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.956166 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl" event={"ID":"1f4cf945-e803-4428-9287-633fb3653273","Type":"ContainerDied","Data":"dc886c86395ab21e9a465371bbf3870ed404b8c7ad6141de0351e124ddfdf3a7"} Nov 03 14:29:35.965860 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.965844 4541 scope.go:122] "RemoveContainer" containerID="40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43" Nov 03 14:29:35.966145 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:29:35.966113 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43\": container with ID starting with 40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43 not found: ID does not exist" containerID="40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43" Nov 03 14:29:35.966246 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.966147 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43"} err="failed to get container status \"40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43\": rpc error: code = NotFound desc = could not find container \"40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43\": container with ID starting with 40b86de6a940e5e22781ffe3ec8182a002fb645f41d893bc21862efe7c0a0c43 not found: ID does not exist" Nov 03 14:29:35.966246 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.966172 4541 scope.go:122] "RemoveContainer" containerID="f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b" Nov 03 14:29:35.977029 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.977004 4541 scope.go:122] "RemoveContainer" containerID="f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b" Nov 03 14:29:35.977510 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:29:35.977491 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b\": container with ID starting with f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b not found: ID does not exist" containerID="f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b" Nov 03 14:29:35.977610 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:35.977518 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b"} err="failed to get container status \"f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b\": rpc error: code = NotFound desc = could not find container \"f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b\": container with ID starting with f398ea17a7e5057ea459b2e26018853ade13c7c6d74ffbdd801355c93f6e801b not found: ID does not exist" Nov 03 14:29:36.110215 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:36.110167 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl"] Nov 03 14:29:36.112541 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:36.112509 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-367381-84d98bdf8f-4xrvl"] Nov 03 14:29:36.214160 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:36.214071 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt"] Nov 03 14:29:36.220093 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:36.220067 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-367381-769b56f99-q4blt"] Nov 03 14:29:36.867904 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:36.867873 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4cf945-e803-4428-9287-633fb3653273" path="/var/lib/kubelet/pods/1f4cf945-e803-4428-9287-633fb3653273/volumes" Nov 03 14:29:36.868291 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:36.868262 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdff9c10-2c0e-4384-a569-099f2d0800ba" path="/var/lib/kubelet/pods/fdff9c10-2c0e-4384-a569-099f2d0800ba/volumes" Nov 03 14:29:37.573871 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:37.573824 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" podUID="fc73c052-3aed-4795-81a4-edfcc8ef1033" containerName="payload-processing" containerID="cri-o://3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353" gracePeriod=25 Nov 03 14:29:37.691578 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:37.691527 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" podUID="8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" containerName="payload-pre-processing" containerID="cri-o://08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3" gracePeriod=25 Nov 03 14:29:38.780612 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:38.780569 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" containerID="cri-o://98f3ede2256abc5547f5105c1889a8773e788709de46768e406f5a57e9e7dda1" gracePeriod=45 Nov 03 14:29:38.799652 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:38.799620 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused" Nov 03 14:29:38.826738 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:38.826699 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" containerName="main" containerID="cri-o://f622d6cf08611f54667880f27bf1322b1b635fce09bf6d151e28a639095b9802" gracePeriod=45 Nov 03 14:29:38.975978 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:38.975941 4541 generic.go:356] "Generic (PLEG): container finished" podID="3660cd0d-de10-4932-8d1e-764392f3e005" containerID="f622d6cf08611f54667880f27bf1322b1b635fce09bf6d151e28a639095b9802" exitCode=0 Nov 03 14:29:38.976165 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:38.976005 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" event={"ID":"3660cd0d-de10-4932-8d1e-764392f3e005","Type":"ContainerDied","Data":"f622d6cf08611f54667880f27bf1322b1b635fce09bf6d151e28a639095b9802"} Nov 03 14:29:38.977868 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:38.977845 4541 generic.go:356] "Generic (PLEG): container finished" podID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerID="98f3ede2256abc5547f5105c1889a8773e788709de46768e406f5a57e9e7dda1" exitCode=0 Nov 03 14:29:38.977979 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:38.977916 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" event={"ID":"582feb39-8aa5-43b6-a68c-6850b5cec3e5","Type":"ContainerDied","Data":"98f3ede2256abc5547f5105c1889a8773e788709de46768e406f5a57e9e7dda1"} Nov 03 14:29:39.082013 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.081974 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 14:29:39.125323 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.125300 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 14:29:39.178052 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178019 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-dshm\") pod \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " Nov 03 14:29:39.178228 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178064 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-home\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-home\") pod \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " Nov 03 14:29:39.178228 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178101 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-model-cache\") pod \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " Nov 03 14:29:39.178228 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178131 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3660cd0d-de10-4932-8d1e-764392f3e005-kube-api-access-9lwtk\" (UniqueName: \"kubernetes.io/projected/3660cd0d-de10-4932-8d1e-764392f3e005-kube-api-access-9lwtk\") pod \"3660cd0d-de10-4932-8d1e-764392f3e005\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " Nov 03 14:29:39.178228 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178189 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tls-certs\") pod \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " Nov 03 14:29:39.178228 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178207 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3660cd0d-de10-4932-8d1e-764392f3e005-tls-certs\" (UniqueName: \"kubernetes.io/secret/3660cd0d-de10-4932-8d1e-764392f3e005-tls-certs\") pod \"3660cd0d-de10-4932-8d1e-764392f3e005\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " Nov 03 14:29:39.178228 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178229 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-home\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-home\") pod \"3660cd0d-de10-4932-8d1e-764392f3e005\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " Nov 03 14:29:39.178548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178266 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-dshm\") pod \"3660cd0d-de10-4932-8d1e-764392f3e005\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " Nov 03 14:29:39.178548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178308 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582feb39-8aa5-43b6-a68c-6850b5cec3e5-kube-api-access-jdmrw\" (UniqueName: \"kubernetes.io/projected/582feb39-8aa5-43b6-a68c-6850b5cec3e5-kube-api-access-jdmrw\") pod \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " Nov 03 14:29:39.178548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178343 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-tmp-dir\") pod \"3660cd0d-de10-4932-8d1e-764392f3e005\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " Nov 03 14:29:39.178548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178372 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tmp-dir\") pod \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\" (UID: \"582feb39-8aa5-43b6-a68c-6850b5cec3e5\") " Nov 03 14:29:39.178548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178394 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-model-cache" pod "582feb39-8aa5-43b6-a68c-6850b5cec3e5" (UID: "582feb39-8aa5-43b6-a68c-6850b5cec3e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.178548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178407 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-model-cache\") pod \"3660cd0d-de10-4932-8d1e-764392f3e005\" (UID: \"3660cd0d-de10-4932-8d1e-764392f3e005\") " Nov 03 14:29:39.178899 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178550 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-home" pod "3660cd0d-de10-4932-8d1e-764392f3e005" (UID: "3660cd0d-de10-4932-8d1e-764392f3e005"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.178899 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178585 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-model-cache" pod "3660cd0d-de10-4932-8d1e-764392f3e005" (UID: "3660cd0d-de10-4932-8d1e-764392f3e005"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.178899 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178802 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-tmp-dir" pod "3660cd0d-de10-4932-8d1e-764392f3e005" (UID: "3660cd0d-de10-4932-8d1e-764392f3e005"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.179071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178960 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-home" pod "582feb39-8aa5-43b6-a68c-6850b5cec3e5" (UID: "582feb39-8aa5-43b6-a68c-6850b5cec3e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.179071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.178978 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tmp-dir" pod "582feb39-8aa5-43b6-a68c-6850b5cec3e5" (UID: "582feb39-8aa5-43b6-a68c-6850b5cec3e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.179071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.179009 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.179071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.179031 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.179071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.179047 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.179071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.179063 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.180705 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.180678 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-dshm" pod "582feb39-8aa5-43b6-a68c-6850b5cec3e5" (UID: "582feb39-8aa5-43b6-a68c-6850b5cec3e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.180858 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.180830 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tls-certs" pod "582feb39-8aa5-43b6-a68c-6850b5cec3e5" (UID: "582feb39-8aa5-43b6-a68c-6850b5cec3e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:39.180962 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.180938 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3660cd0d-de10-4932-8d1e-764392f3e005-kube-api-access-9lwtk" pod "3660cd0d-de10-4932-8d1e-764392f3e005" (UID: "3660cd0d-de10-4932-8d1e-764392f3e005"). InnerVolumeSpecName "kube-api-access-9lwtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:39.181087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.181064 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3660cd0d-de10-4932-8d1e-764392f3e005-tls-certs" pod "3660cd0d-de10-4932-8d1e-764392f3e005" (UID: "3660cd0d-de10-4932-8d1e-764392f3e005"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:39.181211 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.181143 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582feb39-8aa5-43b6-a68c-6850b5cec3e5-kube-api-access-jdmrw" pod "582feb39-8aa5-43b6-a68c-6850b5cec3e5" (UID: "582feb39-8aa5-43b6-a68c-6850b5cec3e5"). InnerVolumeSpecName "kube-api-access-jdmrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:39.181211 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.181198 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-dshm" pod "3660cd0d-de10-4932-8d1e-764392f3e005" (UID: "3660cd0d-de10-4932-8d1e-764392f3e005"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.280306 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.280273 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.280306 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.280303 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lwtk\" (UniqueName: \"kubernetes.io/projected/3660cd0d-de10-4932-8d1e-764392f3e005-kube-api-access-9lwtk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.280506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.280324 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.280506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.280336 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3660cd0d-de10-4932-8d1e-764392f3e005-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.280506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.280346 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3660cd0d-de10-4932-8d1e-764392f3e005-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.280506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.280358 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdmrw\" (UniqueName: \"kubernetes.io/projected/582feb39-8aa5-43b6-a68c-6850b5cec3e5-kube-api-access-jdmrw\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.280506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.280371 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.280506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.280378 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/582feb39-8aa5-43b6-a68c-6850b5cec3e5-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.985516 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.985479 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" Nov 03 14:29:39.985516 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.985499 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt" event={"ID":"582feb39-8aa5-43b6-a68c-6850b5cec3e5","Type":"ContainerDied","Data":"7aa99186b75144580cbaecbe15c8fe5a707cd382b5d0ddbeb3f7853231448b9b"} Nov 03 14:29:39.986115 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.985557 4541 scope.go:122] "RemoveContainer" containerID="98f3ede2256abc5547f5105c1889a8773e788709de46768e406f5a57e9e7dda1" Nov 03 14:29:39.996608 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.992057 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" Nov 03 14:29:39.996608 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.992075 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn" event={"ID":"3660cd0d-de10-4932-8d1e-764392f3e005","Type":"ContainerDied","Data":"f94ca35cc43fd5db651590bd8502f8e4694f004b49bc4be7bbc0cc490d1d8cbd"} Nov 03 14:29:39.996608 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:39.996169 4541 scope.go:122] "RemoveContainer" containerID="f622d6cf08611f54667880f27bf1322b1b635fce09bf6d151e28a639095b9802" Nov 03 14:29:40.141812 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:40.141718 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt"] Nov 03 14:29:40.146400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:40.146369 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-w4-3c5dbc-kserve-f8f96ddb9-t92mt"] Nov 03 14:29:40.250149 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:40.250116 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn"] Nov 03 14:29:40.255498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:40.255471 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/simulator-w4-3c5dbc-kserve-74c7d55d4c-g2wxn"] Nov 03 14:29:40.867413 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:40.867372 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" path="/var/lib/kubelet/pods/3660cd0d-de10-4932-8d1e-764392f3e005/volumes" Nov 03 14:29:40.867905 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:40.867884 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" path="/var/lib/kubelet/pods/582feb39-8aa5-43b6-a68c-6850b5cec3e5/volumes" Nov 03 14:29:45.189523 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:45.189480 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" podUID="8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:29:45.190005 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:45.189494 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" podUID="fc73c052-3aed-4795-81a4-edfcc8ef1033" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:29:46.890908 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:29:46.890865 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:29:55.142453 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:55.142418 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg"] Nov 03 14:29:55.189040 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:55.188993 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" podUID="8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:29:55.189468 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:55.189424 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" podUID="fc73c052-3aed-4795-81a4-edfcc8ef1033" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:29:55.190111 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:55.190087 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp"] Nov 03 14:29:57.833178 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:57.833107 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 14:29:57.931500 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:57.931466 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 14:29:58.007801 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.007757 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc73c052-3aed-4795-81a4-edfcc8ef1033-kube-api-access-vtlmq\" (UniqueName: \"kubernetes.io/projected/fc73c052-3aed-4795-81a4-edfcc8ef1033-kube-api-access-vtlmq\") pod \"fc73c052-3aed-4795-81a4-edfcc8ef1033\" (UID: \"fc73c052-3aed-4795-81a4-edfcc8ef1033\") " Nov 03 14:29:58.007989 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.007896 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc73c052-3aed-4795-81a4-edfcc8ef1033-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73c052-3aed-4795-81a4-edfcc8ef1033-plugins-config-volume\") pod \"fc73c052-3aed-4795-81a4-edfcc8ef1033\" (UID: \"fc73c052-3aed-4795-81a4-edfcc8ef1033\") " Nov 03 14:29:58.008261 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.008236 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc73c052-3aed-4795-81a4-edfcc8ef1033-plugins-config-volume" pod "fc73c052-3aed-4795-81a4-edfcc8ef1033" (UID: "fc73c052-3aed-4795-81a4-edfcc8ef1033"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:58.009824 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.009796 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc73c052-3aed-4795-81a4-edfcc8ef1033-kube-api-access-vtlmq" pod "fc73c052-3aed-4795-81a4-edfcc8ef1033" (UID: "fc73c052-3aed-4795-81a4-edfcc8ef1033"). InnerVolumeSpecName "kube-api-access-vtlmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:58.099443 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.099352 4541 generic.go:356] "Generic (PLEG): container finished" podID="8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" containerID="08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3" exitCode=0 Nov 03 14:29:58.099443 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.099401 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" Nov 03 14:29:58.099687 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.099442 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" event={"ID":"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9","Type":"ContainerDied","Data":"08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3"} Nov 03 14:29:58.099687 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.099500 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh" event={"ID":"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9","Type":"ContainerDied","Data":"02700fd21bf48adaa633e87cdabd58434c604c110b778932d873292a1d1fa640"} Nov 03 14:29:58.099687 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.099532 4541 scope.go:122] "RemoveContainer" containerID="08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3" Nov 03 14:29:58.100796 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.100749 4541 generic.go:356] "Generic (PLEG): container finished" podID="fc73c052-3aed-4795-81a4-edfcc8ef1033" containerID="3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353" exitCode=0 Nov 03 14:29:58.100894 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.100807 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" Nov 03 14:29:58.100894 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.100816 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" event={"ID":"fc73c052-3aed-4795-81a4-edfcc8ef1033","Type":"ContainerDied","Data":"3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353"} Nov 03 14:29:58.100894 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.100870 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p" event={"ID":"fc73c052-3aed-4795-81a4-edfcc8ef1033","Type":"ContainerDied","Data":"022d5b1238aec4b3bd78f4733db10875036b1a4639a3047c6ab4ac261754d4f1"} Nov 03 14:29:58.109270 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.109249 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-kube-api-access-2bfbj\" (UniqueName: \"kubernetes.io/projected/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-kube-api-access-2bfbj\") pod \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\" (UID: \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\") " Nov 03 14:29:58.109391 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.109318 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-plugins-config-volume\") pod \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\" (UID: \"8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9\") " Nov 03 14:29:58.109836 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.109764 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtlmq\" (UniqueName: \"kubernetes.io/projected/fc73c052-3aed-4795-81a4-edfcc8ef1033-kube-api-access-vtlmq\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:58.109836 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.109796 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-plugins-config-volume" pod "8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" (UID: "8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:58.109836 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.109815 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc73c052-3aed-4795-81a4-edfcc8ef1033-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:58.110810 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.110760 4541 scope.go:122] "RemoveContainer" containerID="08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3" Nov 03 14:29:58.111160 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:29:58.111134 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3\": container with ID starting with 08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3 not found: ID does not exist" containerID="08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3" Nov 03 14:29:58.111261 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.111165 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3"} err="failed to get container status \"08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3\": rpc error: code = NotFound desc = could not find container \"08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3\": container with ID starting with 08ab396bd565a5b38630a0d63c5dc7eb9233005b0f0bc5065facb94d9c84cce3 not found: ID does not exist" Nov 03 14:29:58.111261 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.111181 4541 scope.go:122] "RemoveContainer" containerID="3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353" Nov 03 14:29:58.111674 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.111651 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-kube-api-access-2bfbj" pod "8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" (UID: "8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9"). InnerVolumeSpecName "kube-api-access-2bfbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:58.120283 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.120259 4541 scope.go:122] "RemoveContainer" containerID="3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353" Nov 03 14:29:58.120590 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:29:58.120566 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353\": container with ID starting with 3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353 not found: ID does not exist" containerID="3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353" Nov 03 14:29:58.120653 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.120600 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353"} err="failed to get container status \"3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353\": rpc error: code = NotFound desc = could not find container \"3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353\": container with ID starting with 3f7029cdd1f25a49ae8d203f3ccc074c95f6fc3e5f8a4baec741a72361eee353 not found: ID does not exist" Nov 03 14:29:58.210673 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.210639 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bfbj\" (UniqueName: \"kubernetes.io/projected/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-kube-api-access-2bfbj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:58.210673 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.210667 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:58.254303 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.254267 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p"] Nov 03 14:29:58.258344 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.258315 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3c5dbc-5c886bcf5-h8s7p"] Nov 03 14:29:58.529336 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.529300 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh"] Nov 03 14:29:58.541828 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.541797 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3c5dbc-cd4c87467-rkbsh"] Nov 03 14:29:58.867098 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.867021 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" path="/var/lib/kubelet/pods/8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9/volumes" Nov 03 14:29:58.867458 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:29:58.867419 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc73c052-3aed-4795-81a4-edfcc8ef1033" path="/var/lib/kubelet/pods/fc73c052-3aed-4795-81a4-edfcc8ef1033/volumes" Nov 03 14:30:02.501283 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.501240 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5"] Nov 03 14:30:02.502141 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.502076 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" podUID="b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" containerName="maas-api" containerID="cri-o://df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd" gracePeriod=30 Nov 03 14:30:02.754288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.754226 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 14:30:02.850578 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.850539 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-kube-api-access-8qd7m\" (UniqueName: \"kubernetes.io/projected/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-kube-api-access-8qd7m\") pod \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " Nov 03 14:30:02.850811 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.850595 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-metrics-certs\") pod \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " Nov 03 14:30:02.850811 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.850636 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls\") pod \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " Nov 03 14:30:02.850811 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.850676 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-openshift-service-ca\") pod \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\" (UID: \"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383\") " Nov 03 14:30:02.851118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.851091 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-openshift-service-ca" pod "b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" (UID: "b09a6b8d-6afc-4e9a-bb05-53d6b7a67383"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:02.852922 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.852893 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-kube-api-access-8qd7m" pod "b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" (UID: "b09a6b8d-6afc-4e9a-bb05-53d6b7a67383"). InnerVolumeSpecName "kube-api-access-8qd7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:02.853028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.852912 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-metrics-certs" pod "b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" (UID: "b09a6b8d-6afc-4e9a-bb05-53d6b7a67383"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:02.853028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.852932 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls" pod "b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" (UID: "b09a6b8d-6afc-4e9a-bb05-53d6b7a67383"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:02.951678 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.951637 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qd7m\" (UniqueName: \"kubernetes.io/projected/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-kube-api-access-8qd7m\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:02.951678 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.951674 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:02.951678 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.951687 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:02.951936 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:02.951700 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:03.128503 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:03.128413 4541 generic.go:356] "Generic (PLEG): container finished" podID="b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" containerID="df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd" exitCode=0 Nov 03 14:30:03.128503 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:03.128458 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" Nov 03 14:30:03.128732 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:03.128497 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" event={"ID":"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383","Type":"ContainerDied","Data":"df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd"} Nov 03 14:30:03.128732 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:03.128555 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5" event={"ID":"b09a6b8d-6afc-4e9a-bb05-53d6b7a67383","Type":"ContainerDied","Data":"1f1cfd3d0425b6be4b39144baabd6d3c4d6566da68dcc66f09b35fe873718b71"} Nov 03 14:30:03.128732 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:03.128577 4541 scope.go:122] "RemoveContainer" containerID="df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd" Nov 03 14:30:03.137860 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:03.137835 4541 scope.go:122] "RemoveContainer" containerID="df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd" Nov 03 14:30:03.138148 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:30:03.138123 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd\": container with ID starting with df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd not found: ID does not exist" containerID="df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd" Nov 03 14:30:03.138241 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:03.138159 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd"} err="failed to get container status \"df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd\": rpc error: code = NotFound desc = could not find container \"df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd\": container with ID starting with df74f9dc79f492dd2501b883ddb1ef2e5e4704840502a7be384a45f8f31598fd not found: ID does not exist" Nov 03 14:30:03.284930 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:03.284899 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5"] Nov 03 14:30:03.288246 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:03.288193 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-3c5dbc-6bd8fcd588-978p5"] Nov 03 14:30:04.867837 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:04.867808 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" path="/var/lib/kubelet/pods/b09a6b8d-6afc-4e9a-bb05-53d6b7a67383/volumes" Nov 03 14:30:08.934842 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.934808 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk"] Nov 03 14:30:08.935233 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935219 4541 cpu_manager.go:414] "containerMap: removing container" podUID="948aef81-0234-4290-897f-bba6242b630e" containerName="main" Nov 03 14:30:08.935233 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935230 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948aef81-0234-4290-897f-bba6242b630e" containerName="main" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935242 4541 cpu_manager.go:414] "containerMap: removing container" podUID="fdff9c10-2c0e-4384-a569-099f2d0800ba" containerName="payload-pre-processing" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935247 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdff9c10-2c0e-4384-a569-099f2d0800ba" containerName="payload-pre-processing" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935260 4541 cpu_manager.go:414] "containerMap: removing container" podUID="aafbdc79-6bde-48a0-853c-8a92f80481eb" containerName="main" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935266 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafbdc79-6bde-48a0-853c-8a92f80481eb" containerName="main" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935276 4541 cpu_manager.go:414] "containerMap: removing container" podUID="4f83df78-c34c-4c09-b36a-f819e54ddebb" containerName="maas-api" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935280 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f83df78-c34c-4c09-b36a-f819e54ddebb" containerName="maas-api" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935288 4541 cpu_manager.go:414] "containerMap: removing container" podUID="16e8525a-86b2-440e-8215-3431a27a6a54" containerName="payload-processing" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935293 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e8525a-86b2-440e-8215-3431a27a6a54" containerName="payload-processing" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935300 4541 cpu_manager.go:414] "containerMap: removing container" podUID="8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" containerName="payload-pre-processing" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935305 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" containerName="payload-pre-processing" Nov 03 14:30:08.935311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935316 4541 cpu_manager.go:414] "containerMap: removing container" podUID="b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935321 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935327 4541 cpu_manager.go:414] "containerMap: removing container" podUID="cd08615b-547e-4e1d-a572-70f216d10e41" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935331 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd08615b-547e-4e1d-a572-70f216d10e41" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935337 4541 cpu_manager.go:414] "containerMap: removing container" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935341 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935346 4541 cpu_manager.go:414] "containerMap: removing container" podUID="dcea7270-5708-4b70-b0dd-15304574e276" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935352 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcea7270-5708-4b70-b0dd-15304574e276" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935360 4541 cpu_manager.go:414] "containerMap: removing container" podUID="fc73c052-3aed-4795-81a4-edfcc8ef1033" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935365 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc73c052-3aed-4795-81a4-edfcc8ef1033" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935371 4541 cpu_manager.go:414] "containerMap: removing container" podUID="9890570b-67ea-47e3-b607-c1456210c7a6" containerName="payload-pre-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935375 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9890570b-67ea-47e3-b607-c1456210c7a6" containerName="payload-pre-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935384 4541 cpu_manager.go:414] "containerMap: removing container" podUID="a01491e5-7f57-4419-9186-c2966e8cc22f" containerName="payload-pre-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935388 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01491e5-7f57-4419-9186-c2966e8cc22f" containerName="payload-pre-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935394 4541 cpu_manager.go:414] "containerMap: removing container" podUID="256691b8-991d-4105-9b10-40dbcc6dcc2c" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935399 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256691b8-991d-4105-9b10-40dbcc6dcc2c" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935406 4541 cpu_manager.go:414] "containerMap: removing container" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935410 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935416 4541 cpu_manager.go:414] "containerMap: removing container" podUID="ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935421 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935432 4541 cpu_manager.go:414] "containerMap: removing container" podUID="1f4cf945-e803-4428-9287-633fb3653273" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935437 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4cf945-e803-4428-9287-633fb3653273" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935445 4541 cpu_manager.go:414] "containerMap: removing container" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935449 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935455 4541 cpu_manager.go:414] "containerMap: removing container" podUID="b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935459 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935467 4541 cpu_manager.go:414] "containerMap: removing container" podUID="2798c769-768f-4eff-9944-61f6dc97e873" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935472 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2798c769-768f-4eff-9944-61f6dc97e873" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935530 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed40d4b9-4b0f-4500-ae39-1f6b12d49d98" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935542 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4cf945-e803-4428-9287-633fb3653273" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935547 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="3660cd0d-de10-4932-8d1e-764392f3e005" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935554 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04af2a4-f3f3-41ee-a562-1c66dc4bc8b5" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935560 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="2798c769-768f-4eff-9944-61f6dc97e873" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935566 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935571 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdff9c10-2c0e-4384-a569-099f2d0800ba" containerName="payload-pre-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935576 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="948aef81-0234-4290-897f-bba6242b630e" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935582 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafbdc79-6bde-48a0-853c-8a92f80481eb" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935587 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f83df78-c34c-4c09-b36a-f819e54ddebb" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935592 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e8525a-86b2-440e-8215-3431a27a6a54" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935597 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1f7135-4ef0-4ac3-8bc7-ed4fa6aa89e9" containerName="payload-pre-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935604 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="b09a6b8d-6afc-4e9a-bb05-53d6b7a67383" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935609 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd08615b-547e-4e1d-a572-70f216d10e41" containerName="maas-api" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935614 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="582feb39-8aa5-43b6-a68c-6850b5cec3e5" containerName="main" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935619 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcea7270-5708-4b70-b0dd-15304574e276" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935625 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc73c052-3aed-4795-81a4-edfcc8ef1033" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935630 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="9890570b-67ea-47e3-b607-c1456210c7a6" containerName="payload-pre-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935636 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01491e5-7f57-4419-9186-c2966e8cc22f" containerName="payload-pre-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935641 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="256691b8-991d-4105-9b10-40dbcc6dcc2c" containerName="payload-processing" Nov 03 14:30:08.935636 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.935646 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" Nov 03 14:30:08.940356 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.940335 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:08.994240 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:08.994209 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk"] Nov 03 14:30:09.014071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.014002 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9cea6c-vv5nk\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.014273 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.014078 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9cea6c-vv5nk\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.014273 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.014120 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ks2l\" (UniqueName: \"kubernetes.io/projected/3e8d1aab-adc2-4c58-9f30-da34e5d41754-kube-api-access-4ks2l\") pod \"maas-api-revoke-sub-de9cea6c-vv5nk\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.038028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.038003 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:30:09.038174 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.038050 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:30:09.114862 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.114827 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9cea6c-vv5nk\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.115035 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.114870 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9cea6c-vv5nk\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.115035 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.114933 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ks2l\" (UniqueName: \"kubernetes.io/projected/3e8d1aab-adc2-4c58-9f30-da34e5d41754-kube-api-access-4ks2l\") pod \"maas-api-revoke-sub-de9cea6c-vv5nk\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.115655 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.115634 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de9cea6c-vv5nk\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.117322 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.117300 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de9cea6c-vv5nk\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.170870 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.170842 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ks2l\" (UniqueName: \"kubernetes.io/projected/3e8d1aab-adc2-4c58-9f30-da34e5d41754-kube-api-access-4ks2l\") pod \"maas-api-revoke-sub-de9cea6c-vv5nk\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.222548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.222466 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq"] Nov 03 14:30:09.253566 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.253536 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:09.630032 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:09.630002 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk"] Nov 03 14:30:09.632223 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:30:09.632189 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8d1aab_adc2_4c58_9f30_da34e5d41754.slice/crio-c3c86cea5b662155ca4292dea3565cc995cecf31225ca1e9f4dce326008acbf4 WatchSource:0}: Error finding container c3c86cea5b662155ca4292dea3565cc995cecf31225ca1e9f4dce326008acbf4: Status 404 returned error can't find the container with id c3c86cea5b662155ca4292dea3565cc995cecf31225ca1e9f4dce326008acbf4 Nov 03 14:30:10.163097 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.163057 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" podUID="cca463a2-02b8-46a2-8a66-fd398b50eafd" containerName="main" containerID="cri-o://d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872" gracePeriod=45 Nov 03 14:30:10.166427 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.166391 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" event={"ID":"3e8d1aab-adc2-4c58-9f30-da34e5d41754","Type":"ContainerStarted","Data":"6542a225014665d4b8445db376e9ab633fb1ae79b75692321b7a792a907f6153"} Nov 03 14:30:10.166568 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.166467 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" event={"ID":"3e8d1aab-adc2-4c58-9f30-da34e5d41754","Type":"ContainerStarted","Data":"c3c86cea5b662155ca4292dea3565cc995cecf31225ca1e9f4dce326008acbf4"} Nov 03 14:30:10.210188 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.210089 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerName="main" containerID="cri-o://523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818" gracePeriod=45 Nov 03 14:30:10.276701 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.276645 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" podStartSLOduration=2.276631064 podStartE2EDuration="2.276631064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:10.275539998 +0000 UTC m=+5141.589511860" watchObservedRunningTime="2026-11-03 14:30:10.276631064 +0000 UTC m=+5141.590602923" Nov 03 14:30:10.477044 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.476978 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 14:30:10.490753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.490729 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 14:30:10.537473 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537443 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tmp-dir\") pod \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " Nov 03 14:30:10.537657 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537493 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-tmp-dir\") pod \"cca463a2-02b8-46a2-8a66-fd398b50eafd\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " Nov 03 14:30:10.537657 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537546 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-model-cache\") pod \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " Nov 03 14:30:10.537657 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537566 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-kube-api-access-pbpgb\" (UniqueName: \"kubernetes.io/projected/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-kube-api-access-pbpgb\") pod \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " Nov 03 14:30:10.537657 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537613 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-model-cache\") pod \"cca463a2-02b8-46a2-8a66-fd398b50eafd\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " Nov 03 14:30:10.537657 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537642 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tls-certs\" (UniqueName: \"kubernetes.io/secret/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tls-certs\") pod \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " Nov 03 14:30:10.537980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537672 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-dshm\") pod \"cca463a2-02b8-46a2-8a66-fd398b50eafd\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " Nov 03 14:30:10.537980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537712 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca463a2-02b8-46a2-8a66-fd398b50eafd-kube-api-access-nlw5s\" (UniqueName: \"kubernetes.io/projected/cca463a2-02b8-46a2-8a66-fd398b50eafd-kube-api-access-nlw5s\") pod \"cca463a2-02b8-46a2-8a66-fd398b50eafd\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " Nov 03 14:30:10.537980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537761 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tmp-dir" pod "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" (UID: "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.537980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537791 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cca463a2-02b8-46a2-8a66-fd398b50eafd-tls-certs\" (UniqueName: \"kubernetes.io/secret/cca463a2-02b8-46a2-8a66-fd398b50eafd-tls-certs\") pod \"cca463a2-02b8-46a2-8a66-fd398b50eafd\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " Nov 03 14:30:10.537980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537843 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-model-cache" pod "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" (UID: "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.537980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537848 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-tmp-dir" pod "cca463a2-02b8-46a2-8a66-fd398b50eafd" (UID: "cca463a2-02b8-46a2-8a66-fd398b50eafd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.537980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537882 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-dshm\") pod \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " Nov 03 14:30:10.537980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537929 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-home\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-home\") pod \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\" (UID: \"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0\") " Nov 03 14:30:10.537980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537973 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-home\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-home\") pod \"cca463a2-02b8-46a2-8a66-fd398b50eafd\" (UID: \"cca463a2-02b8-46a2-8a66-fd398b50eafd\") " Nov 03 14:30:10.538435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.537883 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-model-cache" pod "cca463a2-02b8-46a2-8a66-fd398b50eafd" (UID: "cca463a2-02b8-46a2-8a66-fd398b50eafd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.538435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.538368 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.538435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.538401 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.538435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.538407 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-home" pod "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" (UID: "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.538435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.538415 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.538698 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.538453 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.538698 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.538598 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-home" pod "cca463a2-02b8-46a2-8a66-fd398b50eafd" (UID: "cca463a2-02b8-46a2-8a66-fd398b50eafd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.540335 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.540302 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cca463a2-02b8-46a2-8a66-fd398b50eafd-tls-certs" pod "cca463a2-02b8-46a2-8a66-fd398b50eafd" (UID: "cca463a2-02b8-46a2-8a66-fd398b50eafd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:10.540454 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.540365 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-dshm" pod "cca463a2-02b8-46a2-8a66-fd398b50eafd" (UID: "cca463a2-02b8-46a2-8a66-fd398b50eafd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.540507 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.540491 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca463a2-02b8-46a2-8a66-fd398b50eafd-kube-api-access-nlw5s" pod "cca463a2-02b8-46a2-8a66-fd398b50eafd" (UID: "cca463a2-02b8-46a2-8a66-fd398b50eafd"). InnerVolumeSpecName "kube-api-access-nlw5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:10.540626 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.540604 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-kube-api-access-pbpgb" pod "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" (UID: "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0"). InnerVolumeSpecName "kube-api-access-pbpgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:10.540689 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.540673 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tls-certs" pod "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" (UID: "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:10.540724 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.540686 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-dshm" pod "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" (UID: "33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:10.639222 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.639185 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.639222 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.639224 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.639436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.639239 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.639436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.639254 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbpgb\" (UniqueName: \"kubernetes.io/projected/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-kube-api-access-pbpgb\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.639436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.639270 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.639436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.639283 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cca463a2-02b8-46a2-8a66-fd398b50eafd-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.639436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.639298 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlw5s\" (UniqueName: \"kubernetes.io/projected/cca463a2-02b8-46a2-8a66-fd398b50eafd-kube-api-access-nlw5s\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:10.639436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:10.639311 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cca463a2-02b8-46a2-8a66-fd398b50eafd-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:11.173031 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.172995 4541 generic.go:356] "Generic (PLEG): container finished" podID="cca463a2-02b8-46a2-8a66-fd398b50eafd" containerID="d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872" exitCode=0 Nov 03 14:30:11.173548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.173045 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" Nov 03 14:30:11.173548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.173080 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" event={"ID":"cca463a2-02b8-46a2-8a66-fd398b50eafd","Type":"ContainerDied","Data":"d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872"} Nov 03 14:30:11.173548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.173125 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" event={"ID":"cca463a2-02b8-46a2-8a66-fd398b50eafd","Type":"ContainerDied","Data":"d0b33e46328e27247bd23c64bb3f35c835d5cdf281059a6785cb301e3ad9092e"} Nov 03 14:30:11.173548 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.173142 4541 scope.go:122] "RemoveContainer" containerID="d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872" Nov 03 14:30:11.174855 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.174832 4541 generic.go:356] "Generic (PLEG): container finished" podID="3e8d1aab-adc2-4c58-9f30-da34e5d41754" containerID="6542a225014665d4b8445db376e9ab633fb1ae79b75692321b7a792a907f6153" exitCode=0 Nov 03 14:30:11.174974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.174922 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" event={"ID":"3e8d1aab-adc2-4c58-9f30-da34e5d41754","Type":"ContainerDied","Data":"6542a225014665d4b8445db376e9ab633fb1ae79b75692321b7a792a907f6153"} Nov 03 14:30:11.176406 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.176384 4541 generic.go:356] "Generic (PLEG): container finished" podID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerID="523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818" exitCode=0 Nov 03 14:30:11.176507 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.176421 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" Nov 03 14:30:11.176507 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.176428 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" event={"ID":"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0","Type":"ContainerDied","Data":"523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818"} Nov 03 14:30:11.176507 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.176457 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp" event={"ID":"33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0","Type":"ContainerDied","Data":"35adb7681b9ae7b8938c9cdd69b39a3340993a6b0a6201a7efa81bb0482aa35b"} Nov 03 14:30:11.183781 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.183744 4541 scope.go:122] "RemoveContainer" containerID="d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872" Nov 03 14:30:11.184079 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:30:11.184059 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872\": container with ID starting with d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872 not found: ID does not exist" containerID="d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872" Nov 03 14:30:11.184131 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.184087 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872"} err="failed to get container status \"d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872\": rpc error: code = NotFound desc = could not find container \"d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872\": container with ID starting with d212e00ad54f171c7e58371fc1e8eb44fb377ac3b119cbb9ef3712c85e386872 not found: ID does not exist" Nov 03 14:30:11.184131 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.184104 4541 scope.go:122] "RemoveContainer" containerID="523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818" Nov 03 14:30:11.192308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.192283 4541 scope.go:122] "RemoveContainer" containerID="523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818" Nov 03 14:30:11.192589 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:30:11.192568 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818\": container with ID starting with 523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818 not found: ID does not exist" containerID="523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818" Nov 03 14:30:11.192649 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.192596 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818"} err="failed to get container status \"523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818\": rpc error: code = NotFound desc = could not find container \"523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818\": container with ID starting with 523b2bc102bc589f189b26619c69995203034a665a0da8307e80bd57dd2a2818 not found: ID does not exist" Nov 03 14:30:11.328461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.328427 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp"] Nov 03 14:30:11.375556 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.375520 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/premium-w1-34b13c-kserve-7dc8c54bd9-7vgpp"] Nov 03 14:30:11.433628 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.433516 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg" podUID="cca463a2-02b8-46a2-8a66-fd398b50eafd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:30:11.443577 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.443546 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg"] Nov 03 14:30:11.448241 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:11.448212 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-34b13c/distinct-2-w1-34b13c-kserve-8cc9dcd67-b85bg"] Nov 03 14:30:12.319912 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.319886 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:12.356209 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.356174 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-xks-ca\") pod \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " Nov 03 14:30:12.356394 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.356340 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8d1aab-adc2-4c58-9f30-da34e5d41754-kube-api-access-4ks2l\" (UniqueName: \"kubernetes.io/projected/3e8d1aab-adc2-4c58-9f30-da34e5d41754-kube-api-access-4ks2l\") pod \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " Nov 03 14:30:12.356459 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.356413 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-openshift-ca\") pod \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\" (UID: \"3e8d1aab-adc2-4c58-9f30-da34e5d41754\") " Nov 03 14:30:12.356761 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.356734 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-openshift-ca" pod "3e8d1aab-adc2-4c58-9f30-da34e5d41754" (UID: "3e8d1aab-adc2-4c58-9f30-da34e5d41754"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:12.358414 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.358385 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-xks-ca" pod "3e8d1aab-adc2-4c58-9f30-da34e5d41754" (UID: "3e8d1aab-adc2-4c58-9f30-da34e5d41754"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:30:12.358516 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.358391 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8d1aab-adc2-4c58-9f30-da34e5d41754-kube-api-access-4ks2l" pod "3e8d1aab-adc2-4c58-9f30-da34e5d41754" (UID: "3e8d1aab-adc2-4c58-9f30-da34e5d41754"). InnerVolumeSpecName "kube-api-access-4ks2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:12.457287 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.457206 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ks2l\" (UniqueName: \"kubernetes.io/projected/3e8d1aab-adc2-4c58-9f30-da34e5d41754-kube-api-access-4ks2l\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:12.457287 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.457237 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-openshift-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:12.457287 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.457248 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e8d1aab-adc2-4c58-9f30-da34e5d41754-maas-api-xks-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:12.867182 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.867092 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" path="/var/lib/kubelet/pods/33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0/volumes" Nov 03 14:30:12.867601 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:12.867579 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca463a2-02b8-46a2-8a66-fd398b50eafd" path="/var/lib/kubelet/pods/cca463a2-02b8-46a2-8a66-fd398b50eafd/volumes" Nov 03 14:30:13.190961 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:13.190927 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" event={"ID":"3e8d1aab-adc2-4c58-9f30-da34e5d41754","Type":"ContainerDied","Data":"c3c86cea5b662155ca4292dea3565cc995cecf31225ca1e9f4dce326008acbf4"} Nov 03 14:30:13.191150 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:13.190975 4541 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c86cea5b662155ca4292dea3565cc995cecf31225ca1e9f4dce326008acbf4" Nov 03 14:30:13.191150 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:13.190979 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk" Nov 03 14:30:13.786466 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:13.786425 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf"] Nov 03 14:30:13.786936 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:13.786662 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" containerID="cri-o://cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c" gracePeriod=30 Nov 03 14:30:13.833462 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:13.833424 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b"] Nov 03 14:30:13.833688 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:13.833647 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" containerID="cri-o://6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf" gracePeriod=30 Nov 03 14:30:14.223398 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.223335 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" podUID="7014bbd1-6465-4805-8c57-2bf4b6d98821" containerName="payload-pre-processing" containerID="cri-o://58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f" gracePeriod=25 Nov 03 14:30:14.428980 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.428940 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:30:14.429246 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.429216 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:30:14.478460 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.478390 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 14:30:14.578417 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.578375 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7014bbd1-6465-4805-8c57-2bf4b6d98821-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7014bbd1-6465-4805-8c57-2bf4b6d98821-plugins-config-volume\") pod \"7014bbd1-6465-4805-8c57-2bf4b6d98821\" (UID: \"7014bbd1-6465-4805-8c57-2bf4b6d98821\") " Nov 03 14:30:14.578603 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.578505 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7014bbd1-6465-4805-8c57-2bf4b6d98821-kube-api-access-tg827\" (UniqueName: \"kubernetes.io/projected/7014bbd1-6465-4805-8c57-2bf4b6d98821-kube-api-access-tg827\") pod \"7014bbd1-6465-4805-8c57-2bf4b6d98821\" (UID: \"7014bbd1-6465-4805-8c57-2bf4b6d98821\") " Nov 03 14:30:14.578753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.578729 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7014bbd1-6465-4805-8c57-2bf4b6d98821-plugins-config-volume" pod "7014bbd1-6465-4805-8c57-2bf4b6d98821" (UID: "7014bbd1-6465-4805-8c57-2bf4b6d98821"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:14.578843 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.578828 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7014bbd1-6465-4805-8c57-2bf4b6d98821-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:14.580709 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.580682 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7014bbd1-6465-4805-8c57-2bf4b6d98821-kube-api-access-tg827" pod "7014bbd1-6465-4805-8c57-2bf4b6d98821" (UID: "7014bbd1-6465-4805-8c57-2bf4b6d98821"). InnerVolumeSpecName "kube-api-access-tg827". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:14.679559 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.679518 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg827\" (UniqueName: \"kubernetes.io/projected/7014bbd1-6465-4805-8c57-2bf4b6d98821-kube-api-access-tg827\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:14.928288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.928247 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:30:14.928647 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:14.928332 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:30:15.202222 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:15.202189 4541 generic.go:356] "Generic (PLEG): container finished" podID="7014bbd1-6465-4805-8c57-2bf4b6d98821" containerID="58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f" exitCode=0 Nov 03 14:30:15.202448 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:15.202243 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" Nov 03 14:30:15.202448 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:15.202253 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" event={"ID":"7014bbd1-6465-4805-8c57-2bf4b6d98821","Type":"ContainerDied","Data":"58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f"} Nov 03 14:30:15.202448 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:15.202294 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq" event={"ID":"7014bbd1-6465-4805-8c57-2bf4b6d98821","Type":"ContainerDied","Data":"cd67250526833798eacc94dad0d4c1efd883eb75c2634043c8123e9d59f8e7df"} Nov 03 14:30:15.202448 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:15.202315 4541 scope.go:122] "RemoveContainer" containerID="58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f" Nov 03 14:30:15.211071 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:15.211049 4541 scope.go:122] "RemoveContainer" containerID="58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f" Nov 03 14:30:15.211351 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:30:15.211322 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f\": container with ID starting with 58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f not found: ID does not exist" containerID="58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f" Nov 03 14:30:15.211434 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:15.211357 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f"} err="failed to get container status \"58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f\": rpc error: code = NotFound desc = could not find container \"58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f\": container with ID starting with 58517f35e1906893e172ee074d1e4dd700e45a495cc8d847b21cb3a8d79bd22f not found: ID does not exist" Nov 03 14:30:15.359482 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:15.359449 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq"] Nov 03 14:30:15.364102 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:15.364071 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-34b13c-84798ffbff-28kkq"] Nov 03 14:30:16.867883 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:16.867849 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7014bbd1-6465-4805-8c57-2bf4b6d98821" path="/var/lib/kubelet/pods/7014bbd1-6465-4805-8c57-2bf4b6d98821/volumes" Nov 03 14:30:16.896865 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:30:16.896831 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:30:19.090990 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.090966 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:30:19.132053 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.132025 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:30:19.224088 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224057 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-data\") pod \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " Nov 03 14:30:19.224334 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224112 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-envoy\") pod \"9303b207-db31-4672-aea3-cb93766856db\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " Nov 03 14:30:19.224334 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224156 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9303b207-db31-4672-aea3-cb93766856db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303b207-db31-4672-aea3-cb93766856db-istiod-ca-cert\") pod \"9303b207-db31-4672-aea3-cb93766856db\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " Nov 03 14:30:19.224334 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224195 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-kube-api-access-n55kj\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-kube-api-access-n55kj\") pod \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " Nov 03 14:30:19.224334 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224240 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-credential-socket\") pod \"9303b207-db31-4672-aea3-cb93766856db\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " Nov 03 14:30:19.224334 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224266 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-token\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-token\") pod \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " Nov 03 14:30:19.224334 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224292 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-socket\") pod \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224391 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-podinfo\") pod \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224396 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-data" pod "4e55ba40-fb4f-4a14-8d11-8226803aa43c" (UID: "4e55ba40-fb4f-4a14-8d11-8226803aa43c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224441 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-istio-token\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-istio-token\") pod \"9303b207-db31-4672-aea3-cb93766856db\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224470 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-envoy\") pod \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224499 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-data\") pod \"9303b207-db31-4672-aea3-cb93766856db\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224522 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istiod-ca-cert\") pod \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224554 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9303b207-db31-4672-aea3-cb93766856db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303b207-db31-4672-aea3-cb93766856db-istio-podinfo\") pod \"9303b207-db31-4672-aea3-cb93766856db\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224585 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-credential-socket\") pod \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224648 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-socket" pod "4e55ba40-fb4f-4a14-8d11-8226803aa43c" (UID: "4e55ba40-fb4f-4a14-8d11-8226803aa43c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224658 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-certs\") pod \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\" (UID: \"4e55ba40-fb4f-4a14-8d11-8226803aa43c\") " Nov 03 14:30:19.224693 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224694 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-certs\") pod \"9303b207-db31-4672-aea3-cb93766856db\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " Nov 03 14:30:19.225296 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224724 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-kube-api-access-dmmzk\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-kube-api-access-dmmzk\") pod \"9303b207-db31-4672-aea3-cb93766856db\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " Nov 03 14:30:19.225296 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.224800 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-socket\") pod \"9303b207-db31-4672-aea3-cb93766856db\" (UID: \"9303b207-db31-4672-aea3-cb93766856db\") " Nov 03 14:30:19.225296 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225039 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9303b207-db31-4672-aea3-cb93766856db-istiod-ca-cert" pod "9303b207-db31-4672-aea3-cb93766856db" (UID: "9303b207-db31-4672-aea3-cb93766856db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:19.225296 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225235 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-credential-socket" pod "9303b207-db31-4672-aea3-cb93766856db" (UID: "9303b207-db31-4672-aea3-cb93766856db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.225296 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225256 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-socket" pod "9303b207-db31-4672-aea3-cb93766856db" (UID: "9303b207-db31-4672-aea3-cb93766856db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.225571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225335 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.225571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225372 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.225571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225388 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9303b207-db31-4672-aea3-cb93766856db-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.225571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225402 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.225571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225416 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.225571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225413 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-credential-socket" pod "4e55ba40-fb4f-4a14-8d11-8226803aa43c" (UID: "4e55ba40-fb4f-4a14-8d11-8226803aa43c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.225571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225432 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-certs" pod "9303b207-db31-4672-aea3-cb93766856db" (UID: "9303b207-db31-4672-aea3-cb93766856db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.226022 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225631 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-certs" pod "4e55ba40-fb4f-4a14-8d11-8226803aa43c" (UID: "4e55ba40-fb4f-4a14-8d11-8226803aa43c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.226022 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225647 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-data" pod "9303b207-db31-4672-aea3-cb93766856db" (UID: "9303b207-db31-4672-aea3-cb93766856db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.226022 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.225837 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istiod-ca-cert" pod "4e55ba40-fb4f-4a14-8d11-8226803aa43c" (UID: "4e55ba40-fb4f-4a14-8d11-8226803aa43c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:19.226677 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.226649 4541 generic.go:356] "Generic (PLEG): container finished" podID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerID="6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf" exitCode=0 Nov 03 14:30:19.226896 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.226734 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" Nov 03 14:30:19.227546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.226733 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" event={"ID":"4e55ba40-fb4f-4a14-8d11-8226803aa43c","Type":"ContainerDied","Data":"6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf"} Nov 03 14:30:19.227546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.227086 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b" event={"ID":"4e55ba40-fb4f-4a14-8d11-8226803aa43c","Type":"ContainerDied","Data":"e8395799b2c4ce917bee6074af57824e760c23bae0207e51dd74ebf54c8f7876"} Nov 03 14:30:19.227546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.227117 4541 scope.go:122] "RemoveContainer" containerID="6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf" Nov 03 14:30:19.227546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.227479 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-podinfo" pod "4e55ba40-fb4f-4a14-8d11-8226803aa43c" (UID: "4e55ba40-fb4f-4a14-8d11-8226803aa43c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:30:19.227883 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.227860 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-istio-token" pod "9303b207-db31-4672-aea3-cb93766856db" (UID: "9303b207-db31-4672-aea3-cb93766856db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:19.228424 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.228399 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-envoy" pod "9303b207-db31-4672-aea3-cb93766856db" (UID: "9303b207-db31-4672-aea3-cb93766856db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.228508 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.228428 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-token" pod "4e55ba40-fb4f-4a14-8d11-8226803aa43c" (UID: "4e55ba40-fb4f-4a14-8d11-8226803aa43c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:19.228941 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.228793 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-kube-api-access-n55kj" pod "4e55ba40-fb4f-4a14-8d11-8226803aa43c" (UID: "4e55ba40-fb4f-4a14-8d11-8226803aa43c"). InnerVolumeSpecName "kube-api-access-n55kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:19.229563 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.229341 4541 generic.go:356] "Generic (PLEG): container finished" podID="9303b207-db31-4672-aea3-cb93766856db" containerID="cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c" exitCode=0 Nov 03 14:30:19.229563 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.229371 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" event={"ID":"9303b207-db31-4672-aea3-cb93766856db","Type":"ContainerDied","Data":"cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c"} Nov 03 14:30:19.229563 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.229414 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" Nov 03 14:30:19.229563 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.229420 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf" event={"ID":"9303b207-db31-4672-aea3-cb93766856db","Type":"ContainerDied","Data":"73c32be8770621810f8c8931acfa3a1382c4d0b6c135f897a578f807adefa3c9"} Nov 03 14:30:19.229875 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.229849 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9303b207-db31-4672-aea3-cb93766856db-istio-podinfo" pod "9303b207-db31-4672-aea3-cb93766856db" (UID: "9303b207-db31-4672-aea3-cb93766856db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:30:19.230015 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.229995 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-kube-api-access-dmmzk" pod "9303b207-db31-4672-aea3-cb93766856db" (UID: "9303b207-db31-4672-aea3-cb93766856db"). InnerVolumeSpecName "kube-api-access-dmmzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:19.230102 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.230084 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-envoy" pod "4e55ba40-fb4f-4a14-8d11-8226803aa43c" (UID: "4e55ba40-fb4f-4a14-8d11-8226803aa43c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:19.238420 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.238386 4541 scope.go:122] "RemoveContainer" containerID="6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf" Nov 03 14:30:19.238729 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:30:19.238701 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf\": container with ID starting with 6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf not found: ID does not exist" containerID="6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf" Nov 03 14:30:19.238867 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.238734 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf"} err="failed to get container status \"6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf\": rpc error: code = NotFound desc = could not find container \"6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf\": container with ID starting with 6af100c7f90438b8c47561b0e42f87ebf4af6940fad6b20f7a657dc24f2b0ebf not found: ID does not exist" Nov 03 14:30:19.238867 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.238762 4541 scope.go:122] "RemoveContainer" containerID="cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c" Nov 03 14:30:19.248408 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.248390 4541 scope.go:122] "RemoveContainer" containerID="cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c" Nov 03 14:30:19.248669 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:30:19.248648 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c\": container with ID starting with cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c not found: ID does not exist" containerID="cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c" Nov 03 14:30:19.248711 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.248677 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c"} err="failed to get container status \"cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c\": rpc error: code = NotFound desc = could not find container \"cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c\": container with ID starting with cbe8a22e891c41f9721d974d8bacb544b88ed338af3cbb36bbecc6d18af8482c not found: ID does not exist" Nov 03 14:30:19.326050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.325966 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.325996 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326010 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326019 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326027 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326035 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e55ba40-fb4f-4a14-8d11-8226803aa43c-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326044 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9303b207-db31-4672-aea3-cb93766856db-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326050 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326053 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326464 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326062 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e55ba40-fb4f-4a14-8d11-8226803aa43c-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326464 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326070 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326464 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326079 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmmzk\" (UniqueName: \"kubernetes.io/projected/9303b207-db31-4672-aea3-cb93766856db-kube-api-access-dmmzk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326464 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326089 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9303b207-db31-4672-aea3-cb93766856db-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.326464 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.326097 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n55kj\" (UniqueName: \"kubernetes.io/projected/4e55ba40-fb4f-4a14-8d11-8226803aa43c-kube-api-access-n55kj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.649912 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.649885 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf"] Nov 03 14:30:19.657970 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.657935 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-b5gdf"] Nov 03 14:30:19.761028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.760985 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b"] Nov 03 14:30:19.764006 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:19.763975 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9gj2b"] Nov 03 14:30:20.866988 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:20.866958 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" path="/var/lib/kubelet/pods/4e55ba40-fb4f-4a14-8d11-8226803aa43c/volumes" Nov 03 14:30:20.867405 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:20.867386 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9303b207-db31-4672-aea3-cb93766856db" path="/var/lib/kubelet/pods/9303b207-db31-4672-aea3-cb93766856db/volumes" Nov 03 14:30:42.029490 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:42.029396 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg"] Nov 03 14:30:46.889433 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:30:46.889401 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:30:47.030575 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:47.030522 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" podUID="af19ec54-5e4a-402c-ad37-82f779d0cfef" containerName="payload-pre-processing" containerID="cri-o://9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7" gracePeriod=25 Nov 03 14:30:56.343506 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:30:56.343458 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" podUID="af19ec54-5e4a-402c-ad37-82f779d0cfef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:31:06.343167 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:06.343132 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" podUID="af19ec54-5e4a-402c-ad37-82f779d0cfef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:31:07.274041 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.274013 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 14:31:07.394916 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.394834 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af19ec54-5e4a-402c-ad37-82f779d0cfef-kube-api-access-wdbxj\" (UniqueName: \"kubernetes.io/projected/af19ec54-5e4a-402c-ad37-82f779d0cfef-kube-api-access-wdbxj\") pod \"af19ec54-5e4a-402c-ad37-82f779d0cfef\" (UID: \"af19ec54-5e4a-402c-ad37-82f779d0cfef\") " Nov 03 14:31:07.394916 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.394911 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af19ec54-5e4a-402c-ad37-82f779d0cfef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af19ec54-5e4a-402c-ad37-82f779d0cfef-plugins-config-volume\") pod \"af19ec54-5e4a-402c-ad37-82f779d0cfef\" (UID: \"af19ec54-5e4a-402c-ad37-82f779d0cfef\") " Nov 03 14:31:07.395312 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.395265 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af19ec54-5e4a-402c-ad37-82f779d0cfef-plugins-config-volume" pod "af19ec54-5e4a-402c-ad37-82f779d0cfef" (UID: "af19ec54-5e4a-402c-ad37-82f779d0cfef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:07.397008 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.396983 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af19ec54-5e4a-402c-ad37-82f779d0cfef-kube-api-access-wdbxj" pod "af19ec54-5e4a-402c-ad37-82f779d0cfef" (UID: "af19ec54-5e4a-402c-ad37-82f779d0cfef"). InnerVolumeSpecName "kube-api-access-wdbxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:07.465528 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.465490 4541 generic.go:356] "Generic (PLEG): container finished" podID="af19ec54-5e4a-402c-ad37-82f779d0cfef" containerID="9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7" exitCode=0 Nov 03 14:31:07.465723 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.465541 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" Nov 03 14:31:07.465723 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.465559 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" event={"ID":"af19ec54-5e4a-402c-ad37-82f779d0cfef","Type":"ContainerDied","Data":"9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7"} Nov 03 14:31:07.465723 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.465610 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg" event={"ID":"af19ec54-5e4a-402c-ad37-82f779d0cfef","Type":"ContainerDied","Data":"8f2d46800f5e58c3319f02b281d862890a6aa6cfdc9d604cc7d9700c786e7cf9"} Nov 03 14:31:07.465723 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.465637 4541 scope.go:122] "RemoveContainer" containerID="9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7" Nov 03 14:31:07.474328 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.474309 4541 scope.go:122] "RemoveContainer" containerID="9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7" Nov 03 14:31:07.474600 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:31:07.474583 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7\": container with ID starting with 9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7 not found: ID does not exist" containerID="9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7" Nov 03 14:31:07.474652 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.474607 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7"} err="failed to get container status \"9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7\": rpc error: code = NotFound desc = could not find container \"9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7\": container with ID starting with 9b905b99c2f2fc4351636e09c6e9ef0f154db504236e487126c927fd3a8f99c7 not found: ID does not exist" Nov 03 14:31:07.496491 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.496457 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdbxj\" (UniqueName: \"kubernetes.io/projected/af19ec54-5e4a-402c-ad37-82f779d0cfef-kube-api-access-wdbxj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:07.496491 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.496487 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af19ec54-5e4a-402c-ad37-82f779d0cfef-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:07.622189 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.622155 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg"] Nov 03 14:31:07.626360 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:07.626330 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2bceb1-7ccf4f75c5-gsgdg"] Nov 03 14:31:08.868520 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:08.868487 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af19ec54-5e4a-402c-ad37-82f779d0cfef" path="/var/lib/kubelet/pods/af19ec54-5e4a-402c-ad37-82f779d0cfef/volumes" Nov 03 14:31:16.890074 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:31:16.890038 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:31:35.988047 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:35.988008 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992"] Nov 03 14:31:35.988610 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:35.988236 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" podUID="4e3e2e04-551b-4094-b4d4-6f35b41069ef" containerName="maas-api" containerID="cri-o://050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751" gracePeriod=30 Nov 03 14:31:36.243458 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.243387 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 14:31:36.363402 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.363362 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e3e2e04-551b-4094-b4d4-6f35b41069ef-kube-api-access-b8sjg\" (UniqueName: \"kubernetes.io/projected/4e3e2e04-551b-4094-b4d4-6f35b41069ef-kube-api-access-b8sjg\") pod \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " Nov 03 14:31:36.363603 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.363478 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls\") pod \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " Nov 03 14:31:36.363603 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.363586 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-metrics-certs\") pod \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " Nov 03 14:31:36.363734 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.363614 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e3e2e04-551b-4094-b4d4-6f35b41069ef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3e2e04-551b-4094-b4d4-6f35b41069ef-openshift-service-ca\") pod \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\" (UID: \"4e3e2e04-551b-4094-b4d4-6f35b41069ef\") " Nov 03 14:31:36.364305 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.364271 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e3e2e04-551b-4094-b4d4-6f35b41069ef-openshift-service-ca" pod "4e3e2e04-551b-4094-b4d4-6f35b41069ef" (UID: "4e3e2e04-551b-4094-b4d4-6f35b41069ef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:36.366654 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.366612 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-metrics-certs" pod "4e3e2e04-551b-4094-b4d4-6f35b41069ef" (UID: "4e3e2e04-551b-4094-b4d4-6f35b41069ef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:36.366814 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.366687 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls" pod "4e3e2e04-551b-4094-b4d4-6f35b41069ef" (UID: "4e3e2e04-551b-4094-b4d4-6f35b41069ef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:36.367087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.367052 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e3e2e04-551b-4094-b4d4-6f35b41069ef-kube-api-access-b8sjg" pod "4e3e2e04-551b-4094-b4d4-6f35b41069ef" (UID: "4e3e2e04-551b-4094-b4d4-6f35b41069ef"). InnerVolumeSpecName "kube-api-access-b8sjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:36.464933 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.464895 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8sjg\" (UniqueName: \"kubernetes.io/projected/4e3e2e04-551b-4094-b4d4-6f35b41069ef-kube-api-access-b8sjg\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:36.464933 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.464924 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:36.464933 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.464934 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e3e2e04-551b-4094-b4d4-6f35b41069ef-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:36.464933 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.464943 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e3e2e04-551b-4094-b4d4-6f35b41069ef-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:36.599595 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.599506 4541 generic.go:356] "Generic (PLEG): container finished" podID="4e3e2e04-551b-4094-b4d4-6f35b41069ef" containerID="050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751" exitCode=0 Nov 03 14:31:36.599595 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.599551 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" Nov 03 14:31:36.599595 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.599565 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" event={"ID":"4e3e2e04-551b-4094-b4d4-6f35b41069ef","Type":"ContainerDied","Data":"050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751"} Nov 03 14:31:36.599953 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.599616 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992" event={"ID":"4e3e2e04-551b-4094-b4d4-6f35b41069ef","Type":"ContainerDied","Data":"10c08befdfac795f574b4164f47f13552c1d5cf18fc16e725cd6365a80d312bd"} Nov 03 14:31:36.599953 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.599642 4541 scope.go:122] "RemoveContainer" containerID="050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751" Nov 03 14:31:36.608807 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.608766 4541 scope.go:122] "RemoveContainer" containerID="050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751" Nov 03 14:31:36.609061 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:31:36.609041 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751\": container with ID starting with 050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751 not found: ID does not exist" containerID="050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751" Nov 03 14:31:36.609108 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.609069 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751"} err="failed to get container status \"050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751\": rpc error: code = NotFound desc = could not find container \"050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751\": container with ID starting with 050dc579bc32fc6e0b03d84efaddfb933867daa7b8c71b9d8f9d8deff29cb751 not found: ID does not exist" Nov 03 14:31:36.754235 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.754200 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992"] Nov 03 14:31:36.801226 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.801183 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-d341af-64c6549c8c-mq992"] Nov 03 14:31:36.866702 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:36.866630 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e3e2e04-551b-4094-b4d4-6f35b41069ef" path="/var/lib/kubelet/pods/4e3e2e04-551b-4094-b4d4-6f35b41069ef/volumes" Nov 03 14:31:46.889408 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:31:46.889370 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:31:49.517487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:49.517449 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv"] Nov 03 14:31:49.564522 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:49.564482 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627"] Nov 03 14:31:57.215683 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:57.215649 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t"] Nov 03 14:31:57.325170 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:31:57.325124 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws"] Nov 03 14:32:02.216485 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:02.216424 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" podUID="3b4e8e41-848f-4bb0-a195-d3a1282908b2" containerName="payload-processing" containerID="cri-o://e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed" gracePeriod=25 Nov 03 14:32:02.326227 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:02.326149 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" podUID="db992848-af0b-4a8d-aa98-711ff341ea78" containerName="payload-pre-processing" containerID="cri-o://6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600" gracePeriod=25 Nov 03 14:32:04.537269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.537223 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerName="main" containerID="cri-o://47d200e2ee3aa4db69f52b70d45baf4d9ddefe803897dd0ae60a9c62f60ed08e" gracePeriod=45 Nov 03 14:32:04.584879 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.584824 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" podUID="d6edf898-5a79-4539-a61a-ea6a379c98d5" containerName="main" containerID="cri-o://b23e9d7ad87bb86d1d97cea410a78d17316bac6509f7d4b95a59ecf329653c7f" gracePeriod=45 Nov 03 14:32:04.734968 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.732910 4541 generic.go:356] "Generic (PLEG): container finished" podID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerID="47d200e2ee3aa4db69f52b70d45baf4d9ddefe803897dd0ae60a9c62f60ed08e" exitCode=0 Nov 03 14:32:04.734968 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.733036 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" event={"ID":"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b","Type":"ContainerDied","Data":"47d200e2ee3aa4db69f52b70d45baf4d9ddefe803897dd0ae60a9c62f60ed08e"} Nov 03 14:32:04.734968 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.734256 4541 generic.go:356] "Generic (PLEG): container finished" podID="d6edf898-5a79-4539-a61a-ea6a379c98d5" containerID="b23e9d7ad87bb86d1d97cea410a78d17316bac6509f7d4b95a59ecf329653c7f" exitCode=0 Nov 03 14:32:04.734968 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.734329 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" event={"ID":"d6edf898-5a79-4539-a61a-ea6a379c98d5","Type":"ContainerDied","Data":"b23e9d7ad87bb86d1d97cea410a78d17316bac6509f7d4b95a59ecf329653c7f"} Nov 03 14:32:04.846120 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.846093 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 14:32:04.885925 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.885896 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 14:32:04.942494 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.942462 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-model-cache\") pod \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " Nov 03 14:32:04.942687 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.942524 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-dshm\") pod \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " Nov 03 14:32:04.942687 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.942568 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-home\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-home\") pod \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " Nov 03 14:32:04.942687 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.942606 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-kube-api-access-5js2b\" (UniqueName: \"kubernetes.io/projected/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-kube-api-access-5js2b\") pod \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " Nov 03 14:32:04.942930 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.942900 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-model-cache" pod "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" (UID: "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.943068 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.943041 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-home" pod "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" (UID: "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.944093 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.944064 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tmp-dir\") pod \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " Nov 03 14:32:04.944233 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.944100 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tls-certs\") pod \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\" (UID: \"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b\") " Nov 03 14:32:04.944492 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.944348 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tmp-dir" pod "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" (UID: "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.945089 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.945066 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:04.945190 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.945094 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:04.945190 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.945134 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:04.945190 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.945155 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-dshm" pod "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" (UID: "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:04.945523 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.945499 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-kube-api-access-5js2b" pod "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" (UID: "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b"). InnerVolumeSpecName "kube-api-access-5js2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:04.946397 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:04.946369 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tls-certs" pod "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" (UID: "d74e0ba8-a137-4e34-9a7d-a7c4559fce1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:05.046276 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046237 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-dshm\") pod \"d6edf898-5a79-4539-a61a-ea6a379c98d5\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " Nov 03 14:32:05.046470 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046289 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-home\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-home\") pod \"d6edf898-5a79-4539-a61a-ea6a379c98d5\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " Nov 03 14:32:05.046470 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046341 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-tmp-dir\") pod \"d6edf898-5a79-4539-a61a-ea6a379c98d5\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " Nov 03 14:32:05.046597 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046472 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-model-cache\") pod \"d6edf898-5a79-4539-a61a-ea6a379c98d5\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " Nov 03 14:32:05.046597 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046525 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6edf898-5a79-4539-a61a-ea6a379c98d5-kube-api-access-vtrck\" (UniqueName: \"kubernetes.io/projected/d6edf898-5a79-4539-a61a-ea6a379c98d5-kube-api-access-vtrck\") pod \"d6edf898-5a79-4539-a61a-ea6a379c98d5\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " Nov 03 14:32:05.046597 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046566 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6edf898-5a79-4539-a61a-ea6a379c98d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edf898-5a79-4539-a61a-ea6a379c98d5-tls-certs\") pod \"d6edf898-5a79-4539-a61a-ea6a379c98d5\" (UID: \"d6edf898-5a79-4539-a61a-ea6a379c98d5\") " Nov 03 14:32:05.046766 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046620 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-home" pod "d6edf898-5a79-4539-a61a-ea6a379c98d5" (UID: "d6edf898-5a79-4539-a61a-ea6a379c98d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:05.046766 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046638 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-tmp-dir" pod "d6edf898-5a79-4539-a61a-ea6a379c98d5" (UID: "d6edf898-5a79-4539-a61a-ea6a379c98d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:05.046906 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046868 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-model-cache" pod "d6edf898-5a79-4539-a61a-ea6a379c98d5" (UID: "d6edf898-5a79-4539-a61a-ea6a379c98d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:05.046966 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046911 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.046966 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046935 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.046966 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046948 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.047118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046965 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5js2b\" (UniqueName: \"kubernetes.io/projected/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-kube-api-access-5js2b\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.047118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.046981 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.049692 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.049660 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6edf898-5a79-4539-a61a-ea6a379c98d5-tls-certs" pod "d6edf898-5a79-4539-a61a-ea6a379c98d5" (UID: "d6edf898-5a79-4539-a61a-ea6a379c98d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:05.049851 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.049812 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6edf898-5a79-4539-a61a-ea6a379c98d5-kube-api-access-vtrck" pod "d6edf898-5a79-4539-a61a-ea6a379c98d5" (UID: "d6edf898-5a79-4539-a61a-ea6a379c98d5"). InnerVolumeSpecName "kube-api-access-vtrck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:05.050021 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.049997 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-dshm" pod "d6edf898-5a79-4539-a61a-ea6a379c98d5" (UID: "d6edf898-5a79-4539-a61a-ea6a379c98d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:05.147503 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.147408 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.147503 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.147444 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtrck\" (UniqueName: \"kubernetes.io/projected/d6edf898-5a79-4539-a61a-ea6a379c98d5-kube-api-access-vtrck\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.147503 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.147460 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6edf898-5a79-4539-a61a-ea6a379c98d5-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.147503 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.147472 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6edf898-5a79-4539-a61a-ea6a379c98d5-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:05.740203 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.740174 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" Nov 03 14:32:05.740668 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.740218 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627" event={"ID":"d6edf898-5a79-4539-a61a-ea6a379c98d5","Type":"ContainerDied","Data":"a8f89cca2e60b08cbec29f360656ac5c9cca424e46d7c735d2cb2e49a241c8a7"} Nov 03 14:32:05.740668 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.740279 4541 scope.go:122] "RemoveContainer" containerID="b23e9d7ad87bb86d1d97cea410a78d17316bac6509f7d4b95a59ecf329653c7f" Nov 03 14:32:05.741641 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.741622 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" Nov 03 14:32:05.741701 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.741647 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv" event={"ID":"d74e0ba8-a137-4e34-9a7d-a7c4559fce1b","Type":"ContainerDied","Data":"a8ba9143a544f220990995cc4af0081c63f6d2f9abe740012efe66c113f84a90"} Nov 03 14:32:05.749281 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.749256 4541 scope.go:122] "RemoveContainer" containerID="47d200e2ee3aa4db69f52b70d45baf4d9ddefe803897dd0ae60a9c62f60ed08e" Nov 03 14:32:05.895516 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.895437 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv"] Nov 03 14:32:05.942398 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:05.942362 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/premium-w2-cdb532-kserve-688f9985bf-jf8qv"] Nov 03 14:32:06.061616 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:06.061570 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627"] Nov 03 14:32:06.067444 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:06.067409 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-cdb532/simulator-w2-cdb532-kserve-5c5d7df75b-gn627"] Nov 03 14:32:06.136513 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:06.136473 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" podUID="3b4e8e41-848f-4bb0-a195-d3a1282908b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:32:06.866602 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:06.866568 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6edf898-5a79-4539-a61a-ea6a379c98d5" path="/var/lib/kubelet/pods/d6edf898-5a79-4539-a61a-ea6a379c98d5/volumes" Nov 03 14:32:06.867009 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:06.866964 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" path="/var/lib/kubelet/pods/d74e0ba8-a137-4e34-9a7d-a7c4559fce1b/volumes" Nov 03 14:32:07.142039 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:07.141956 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" podUID="db992848-af0b-4a8d-aa98-711ff341ea78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:32:08.888831 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:08.888795 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s"] Nov 03 14:32:08.889593 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:08.888980 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" podUID="1045d728-1c5d-49bf-9459-633d5d11aee3" containerName="maas-api" containerID="cri-o://be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677" gracePeriod=30 Nov 03 14:32:09.140608 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.140541 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 14:32:09.191928 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.191891 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls\") pod \"1045d728-1c5d-49bf-9459-633d5d11aee3\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " Nov 03 14:32:09.192095 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.191973 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1045d728-1c5d-49bf-9459-633d5d11aee3-kube-api-access-47m8v\" (UniqueName: \"kubernetes.io/projected/1045d728-1c5d-49bf-9459-633d5d11aee3-kube-api-access-47m8v\") pod \"1045d728-1c5d-49bf-9459-633d5d11aee3\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " Nov 03 14:32:09.192095 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.192029 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1045d728-1c5d-49bf-9459-633d5d11aee3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1045d728-1c5d-49bf-9459-633d5d11aee3-openshift-service-ca\") pod \"1045d728-1c5d-49bf-9459-633d5d11aee3\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " Nov 03 14:32:09.192095 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.192067 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-metrics-certs\") pod \"1045d728-1c5d-49bf-9459-633d5d11aee3\" (UID: \"1045d728-1c5d-49bf-9459-633d5d11aee3\") " Nov 03 14:32:09.192485 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.192450 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1045d728-1c5d-49bf-9459-633d5d11aee3-openshift-service-ca" pod "1045d728-1c5d-49bf-9459-633d5d11aee3" (UID: "1045d728-1c5d-49bf-9459-633d5d11aee3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:09.194182 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.194148 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-metrics-certs" pod "1045d728-1c5d-49bf-9459-633d5d11aee3" (UID: "1045d728-1c5d-49bf-9459-633d5d11aee3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:09.194294 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.194204 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls" pod "1045d728-1c5d-49bf-9459-633d5d11aee3" (UID: "1045d728-1c5d-49bf-9459-633d5d11aee3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:09.194294 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.194206 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1045d728-1c5d-49bf-9459-633d5d11aee3-kube-api-access-47m8v" pod "1045d728-1c5d-49bf-9459-633d5d11aee3" (UID: "1045d728-1c5d-49bf-9459-633d5d11aee3"). InnerVolumeSpecName "kube-api-access-47m8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:09.293237 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.293206 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47m8v\" (UniqueName: \"kubernetes.io/projected/1045d728-1c5d-49bf-9459-633d5d11aee3-kube-api-access-47m8v\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:09.293237 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.293234 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1045d728-1c5d-49bf-9459-633d5d11aee3-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:09.293456 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.293245 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:09.293456 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.293255 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1045d728-1c5d-49bf-9459-633d5d11aee3-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:09.763199 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.763161 4541 generic.go:356] "Generic (PLEG): container finished" podID="1045d728-1c5d-49bf-9459-633d5d11aee3" containerID="be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677" exitCode=0 Nov 03 14:32:09.763378 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.763207 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" Nov 03 14:32:09.763378 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.763208 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" event={"ID":"1045d728-1c5d-49bf-9459-633d5d11aee3","Type":"ContainerDied","Data":"be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677"} Nov 03 14:32:09.763378 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.763366 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s" event={"ID":"1045d728-1c5d-49bf-9459-633d5d11aee3","Type":"ContainerDied","Data":"2e0cc89427b01253b55b2ebef42b7d3012a622740ec78a2cd6b22eac3d977478"} Nov 03 14:32:09.763549 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.763400 4541 scope.go:122] "RemoveContainer" containerID="be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677" Nov 03 14:32:09.772591 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.772573 4541 scope.go:122] "RemoveContainer" containerID="be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677" Nov 03 14:32:09.772883 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:32:09.772860 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677\": container with ID starting with be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677 not found: ID does not exist" containerID="be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677" Nov 03 14:32:09.772929 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.772893 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677"} err="failed to get container status \"be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677\": rpc error: code = NotFound desc = could not find container \"be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677\": container with ID starting with be1cc6c90e325f483ae293a203f86bfd3b1b2578e3a4bea356acc5087ece4677 not found: ID does not exist" Nov 03 14:32:09.917710 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.917675 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s"] Nov 03 14:32:09.964844 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:09.964748 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-cdb532-596cd4cb45-q7d5s"] Nov 03 14:32:10.867558 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:10.867525 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1045d728-1c5d-49bf-9459-633d5d11aee3" path="/var/lib/kubelet/pods/1045d728-1c5d-49bf-9459-633d5d11aee3/volumes" Nov 03 14:32:16.136251 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:16.136209 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" podUID="3b4e8e41-848f-4bb0-a195-d3a1282908b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:32:16.889280 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:32:16.889234 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:32:17.141553 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:17.141471 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" podUID="db992848-af0b-4a8d-aa98-711ff341ea78" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:32:22.461990 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.461962 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 14:32:22.505547 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.505258 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4e8e41-848f-4bb0-a195-d3a1282908b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4e8e41-848f-4bb0-a195-d3a1282908b2-plugins-config-volume\") pod \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\" (UID: \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\") " Nov 03 14:32:22.505547 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.505338 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4e8e41-848f-4bb0-a195-d3a1282908b2-kube-api-access-lql4g\" (UniqueName: \"kubernetes.io/projected/3b4e8e41-848f-4bb0-a195-d3a1282908b2-kube-api-access-lql4g\") pod \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\" (UID: \"3b4e8e41-848f-4bb0-a195-d3a1282908b2\") " Nov 03 14:32:22.505734 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.505632 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4e8e41-848f-4bb0-a195-d3a1282908b2-plugins-config-volume" pod "3b4e8e41-848f-4bb0-a195-d3a1282908b2" (UID: "3b4e8e41-848f-4bb0-a195-d3a1282908b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:22.507803 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.507779 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4e8e41-848f-4bb0-a195-d3a1282908b2-kube-api-access-lql4g" pod "3b4e8e41-848f-4bb0-a195-d3a1282908b2" (UID: "3b4e8e41-848f-4bb0-a195-d3a1282908b2"). InnerVolumeSpecName "kube-api-access-lql4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:22.564175 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.564148 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 14:32:22.606046 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.606012 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db992848-af0b-4a8d-aa98-711ff341ea78-kube-api-access-tkw2c\" (UniqueName: \"kubernetes.io/projected/db992848-af0b-4a8d-aa98-711ff341ea78-kube-api-access-tkw2c\") pod \"db992848-af0b-4a8d-aa98-711ff341ea78\" (UID: \"db992848-af0b-4a8d-aa98-711ff341ea78\") " Nov 03 14:32:22.606231 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.606052 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db992848-af0b-4a8d-aa98-711ff341ea78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db992848-af0b-4a8d-aa98-711ff341ea78-plugins-config-volume\") pod \"db992848-af0b-4a8d-aa98-711ff341ea78\" (UID: \"db992848-af0b-4a8d-aa98-711ff341ea78\") " Nov 03 14:32:22.606403 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.606384 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4e8e41-848f-4bb0-a195-d3a1282908b2-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:22.606468 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.606409 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lql4g\" (UniqueName: \"kubernetes.io/projected/3b4e8e41-848f-4bb0-a195-d3a1282908b2-kube-api-access-lql4g\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:22.606468 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.606422 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db992848-af0b-4a8d-aa98-711ff341ea78-plugins-config-volume" pod "db992848-af0b-4a8d-aa98-711ff341ea78" (UID: "db992848-af0b-4a8d-aa98-711ff341ea78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:22.608087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.608067 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db992848-af0b-4a8d-aa98-711ff341ea78-kube-api-access-tkw2c" pod "db992848-af0b-4a8d-aa98-711ff341ea78" (UID: "db992848-af0b-4a8d-aa98-711ff341ea78"). InnerVolumeSpecName "kube-api-access-tkw2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:22.707263 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.707230 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkw2c\" (UniqueName: \"kubernetes.io/projected/db992848-af0b-4a8d-aa98-711ff341ea78-kube-api-access-tkw2c\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:22.707263 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.707262 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db992848-af0b-4a8d-aa98-711ff341ea78-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:22.823605 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.823508 4541 generic.go:356] "Generic (PLEG): container finished" podID="db992848-af0b-4a8d-aa98-711ff341ea78" containerID="6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600" exitCode=0 Nov 03 14:32:22.823605 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.823550 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" Nov 03 14:32:22.823865 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.823605 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" event={"ID":"db992848-af0b-4a8d-aa98-711ff341ea78","Type":"ContainerDied","Data":"6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600"} Nov 03 14:32:22.823865 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.823666 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws" event={"ID":"db992848-af0b-4a8d-aa98-711ff341ea78","Type":"ContainerDied","Data":"28dbf344b4dc76a95f593b11018a4f1a5fea9c3a9c76b6671ff3df98c2699ed9"} Nov 03 14:32:22.823865 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.823689 4541 scope.go:122] "RemoveContainer" containerID="6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600" Nov 03 14:32:22.824949 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.824926 4541 generic.go:356] "Generic (PLEG): container finished" podID="3b4e8e41-848f-4bb0-a195-d3a1282908b2" containerID="e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed" exitCode=0 Nov 03 14:32:22.825025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.824962 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" event={"ID":"3b4e8e41-848f-4bb0-a195-d3a1282908b2","Type":"ContainerDied","Data":"e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed"} Nov 03 14:32:22.825025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.824984 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" Nov 03 14:32:22.825025 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.825005 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t" event={"ID":"3b4e8e41-848f-4bb0-a195-d3a1282908b2","Type":"ContainerDied","Data":"fc62bce7e8b8a37b55cdfc1edb74181cfbde9b45275c45b73483344eae363274"} Nov 03 14:32:22.832053 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.832036 4541 scope.go:122] "RemoveContainer" containerID="6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600" Nov 03 14:32:22.832335 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:32:22.832297 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600\": container with ID starting with 6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600 not found: ID does not exist" containerID="6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600" Nov 03 14:32:22.832453 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.832341 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600"} err="failed to get container status \"6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600\": rpc error: code = NotFound desc = could not find container \"6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600\": container with ID starting with 6699b1b5219d60eace3494d10ec795f43f8e86aeae62d6498ddcb6aae04e8600 not found: ID does not exist" Nov 03 14:32:22.832453 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.832363 4541 scope.go:122] "RemoveContainer" containerID="e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed" Nov 03 14:32:22.840834 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.840816 4541 scope.go:122] "RemoveContainer" containerID="e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed" Nov 03 14:32:22.841124 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:32:22.841105 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed\": container with ID starting with e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed not found: ID does not exist" containerID="e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed" Nov 03 14:32:22.841192 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.841135 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed"} err="failed to get container status \"e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed\": rpc error: code = NotFound desc = could not find container \"e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed\": container with ID starting with e379f85e2f5970f5dc5c7cf3c14ea3126fac74541c18dc47c6e21fed259842ed not found: ID does not exist" Nov 03 14:32:22.980687 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.980647 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t"] Nov 03 14:32:22.985560 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:22.985527 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-cdb532-65f9ddfcd5-mn45t"] Nov 03 14:32:23.090232 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:23.090148 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws"] Nov 03 14:32:23.093562 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:23.093531 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-cdb532-5b9b5f8b79-fn6ws"] Nov 03 14:32:24.868722 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:24.868688 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4e8e41-848f-4bb0-a195-d3a1282908b2" path="/var/lib/kubelet/pods/3b4e8e41-848f-4bb0-a195-d3a1282908b2/volumes" Nov 03 14:32:24.869111 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:24.869068 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db992848-af0b-4a8d-aa98-711ff341ea78" path="/var/lib/kubelet/pods/db992848-af0b-4a8d-aa98-711ff341ea78/volumes" Nov 03 14:32:30.599543 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:30.599506 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8"] Nov 03 14:32:30.600065 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:30.599757 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" containerID="cri-o://e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8" gracePeriod=30 Nov 03 14:32:31.747805 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:31.747755 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f5d89c4f6-jtgnc_cef8280b-0119-4e61-94c5-5ddea15d4f04/authorino/0.log" Nov 03 14:32:32.827643 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:32.827596 4541 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:32:32.828058 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:32.827665 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:32:34.736206 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:34.736176 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cb4ffdf47-czmlj_bb24765f-1472-41e4-863d-ecdcc9f1ee97/manager/0.log" Nov 03 14:32:35.849816 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.849791 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 14:32:35.883519 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.883432 4541 generic.go:356] "Generic (PLEG): container finished" podID="b07c5e28-99e1-470a-a007-889ba3917657" containerID="e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8" exitCode=0 Nov 03 14:32:35.883519 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.883476 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" Nov 03 14:32:35.883519 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.883506 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" event={"ID":"b07c5e28-99e1-470a-a007-889ba3917657","Type":"ContainerDied","Data":"e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8"} Nov 03 14:32:35.883747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.883546 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8" event={"ID":"b07c5e28-99e1-470a-a007-889ba3917657","Type":"ContainerDied","Data":"3cb3a94ba98111e7ff2041729254892010f75eea2b768c5f76b60e3d98437bdd"} Nov 03 14:32:35.883747 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.883586 4541 scope.go:122] "RemoveContainer" containerID="e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8" Nov 03 14:32:35.892227 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.892208 4541 scope.go:122] "RemoveContainer" containerID="e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8" Nov 03 14:32:35.892534 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:32:35.892507 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8\": container with ID starting with e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8 not found: ID does not exist" containerID="e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8" Nov 03 14:32:35.892608 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.892546 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8"} err="failed to get container status \"e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8\": rpc error: code = NotFound desc = could not find container \"e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8\": container with ID starting with e8c7b9d14a4147cef9303f1cae577a77f5779246af84b8b88e6eda528402bac8 not found: ID does not exist" Nov 03 14:32:35.930273 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930237 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b07c5e28-99e1-470a-a007-889ba3917657-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b07c5e28-99e1-470a-a007-889ba3917657-istio-podinfo\") pod \"b07c5e28-99e1-470a-a007-889ba3917657\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " Nov 03 14:32:35.930484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930291 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-certs\") pod \"b07c5e28-99e1-470a-a007-889ba3917657\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " Nov 03 14:32:35.930484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930318 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-socket\") pod \"b07c5e28-99e1-470a-a007-889ba3917657\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " Nov 03 14:32:35.930484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930413 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-credential-socket\") pod \"b07c5e28-99e1-470a-a007-889ba3917657\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " Nov 03 14:32:35.930667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930495 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07c5e28-99e1-470a-a007-889ba3917657-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b07c5e28-99e1-470a-a007-889ba3917657-istiod-ca-cert\") pod \"b07c5e28-99e1-470a-a007-889ba3917657\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " Nov 03 14:32:35.930667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930552 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-certs" pod "b07c5e28-99e1-470a-a007-889ba3917657" (UID: "b07c5e28-99e1-470a-a007-889ba3917657"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:35.930667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930555 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-data\") pod \"b07c5e28-99e1-470a-a007-889ba3917657\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " Nov 03 14:32:35.930667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930609 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-socket" pod "b07c5e28-99e1-470a-a007-889ba3917657" (UID: "b07c5e28-99e1-470a-a007-889ba3917657"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:35.930667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930630 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-istio-token\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-istio-token\") pod \"b07c5e28-99e1-470a-a007-889ba3917657\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " Nov 03 14:32:35.931028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930679 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-kube-api-access-bbfsl\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-kube-api-access-bbfsl\") pod \"b07c5e28-99e1-470a-a007-889ba3917657\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " Nov 03 14:32:35.931028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930743 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-credential-socket" pod "b07c5e28-99e1-470a-a007-889ba3917657" (UID: "b07c5e28-99e1-470a-a007-889ba3917657"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:35.931028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930733 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-envoy\") pod \"b07c5e28-99e1-470a-a007-889ba3917657\" (UID: \"b07c5e28-99e1-470a-a007-889ba3917657\") " Nov 03 14:32:35.931028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930762 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-data" pod "b07c5e28-99e1-470a-a007-889ba3917657" (UID: "b07c5e28-99e1-470a-a007-889ba3917657"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:35.931028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.930873 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07c5e28-99e1-470a-a007-889ba3917657-istiod-ca-cert" pod "b07c5e28-99e1-470a-a007-889ba3917657" (UID: "b07c5e28-99e1-470a-a007-889ba3917657"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:35.931282 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.931096 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:35.931282 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.931116 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:35.931282 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.931133 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:35.931282 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.931157 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b07c5e28-99e1-470a-a007-889ba3917657-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:35.931282 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.931170 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:35.932630 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.932602 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b07c5e28-99e1-470a-a007-889ba3917657-istio-podinfo" pod "b07c5e28-99e1-470a-a007-889ba3917657" (UID: "b07c5e28-99e1-470a-a007-889ba3917657"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:32:35.932920 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.932894 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-envoy" pod "b07c5e28-99e1-470a-a007-889ba3917657" (UID: "b07c5e28-99e1-470a-a007-889ba3917657"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:35.932920 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.932897 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-istio-token" pod "b07c5e28-99e1-470a-a007-889ba3917657" (UID: "b07c5e28-99e1-470a-a007-889ba3917657"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:35.933053 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:35.933010 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-kube-api-access-bbfsl" pod "b07c5e28-99e1-470a-a007-889ba3917657" (UID: "b07c5e28-99e1-470a-a007-889ba3917657"). InnerVolumeSpecName "kube-api-access-bbfsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:36.031624 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:36.031585 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:36.031624 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:36.031624 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbfsl\" (UniqueName: \"kubernetes.io/projected/b07c5e28-99e1-470a-a007-889ba3917657-kube-api-access-bbfsl\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:36.031847 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:36.031635 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b07c5e28-99e1-470a-a007-889ba3917657-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:36.031847 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:36.031644 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b07c5e28-99e1-470a-a007-889ba3917657-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:36.298486 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:36.298448 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8"] Nov 03 14:32:36.303352 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:36.303326 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-x96d8"] Nov 03 14:32:36.869291 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:36.869252 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07c5e28-99e1-470a-a007-889ba3917657" path="/var/lib/kubelet/pods/b07c5e28-99e1-470a-a007-889ba3917657/volumes" Nov 03 14:32:46.889720 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:32:46.889679 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:32:49.868347 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868305 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls"] Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868668 4541 cpu_manager.go:414] "containerMap: removing container" podUID="3e8d1aab-adc2-4c58-9f30-da34e5d41754" containerName="revoke-keys" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868682 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8d1aab-adc2-4c58-9f30-da34e5d41754" containerName="revoke-keys" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868693 4541 cpu_manager.go:414] "containerMap: removing container" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868698 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e185d31d-49f8-4a83-a724-5b2219356e3d" containerName="istio-proxy" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868705 4541 cpu_manager.go:414] "containerMap: removing container" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868710 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868726 4541 cpu_manager.go:414] "containerMap: removing container" podUID="1045d728-1c5d-49bf-9459-633d5d11aee3" containerName="maas-api" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868731 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1045d728-1c5d-49bf-9459-633d5d11aee3" containerName="maas-api" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868739 4541 cpu_manager.go:414] "containerMap: removing container" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerName="main" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868743 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerName="main" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868754 4541 cpu_manager.go:414] "containerMap: removing container" podUID="4e3e2e04-551b-4094-b4d4-6f35b41069ef" containerName="maas-api" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868758 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e3e2e04-551b-4094-b4d4-6f35b41069ef" containerName="maas-api" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868766 4541 cpu_manager.go:414] "containerMap: removing container" podUID="d6edf898-5a79-4539-a61a-ea6a379c98d5" containerName="main" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868785 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6edf898-5a79-4539-a61a-ea6a379c98d5" containerName="main" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868792 4541 cpu_manager.go:414] "containerMap: removing container" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868798 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868806 4541 cpu_manager.go:414] "containerMap: removing container" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868810 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868820 4541 cpu_manager.go:414] "containerMap: removing container" podUID="3b4e8e41-848f-4bb0-a195-d3a1282908b2" containerName="payload-processing" Nov 03 14:32:49.868821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868824 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4e8e41-848f-4bb0-a195-d3a1282908b2" containerName="payload-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868834 4541 cpu_manager.go:414] "containerMap: removing container" podUID="af19ec54-5e4a-402c-ad37-82f779d0cfef" containerName="payload-pre-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868839 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af19ec54-5e4a-402c-ad37-82f779d0cfef" containerName="payload-pre-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868850 4541 cpu_manager.go:414] "containerMap: removing container" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerName="main" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868855 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerName="main" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868861 4541 cpu_manager.go:414] "containerMap: removing container" podUID="cca463a2-02b8-46a2-8a66-fd398b50eafd" containerName="main" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868865 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca463a2-02b8-46a2-8a66-fd398b50eafd" containerName="main" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868880 4541 cpu_manager.go:414] "containerMap: removing container" podUID="db992848-af0b-4a8d-aa98-711ff341ea78" containerName="payload-pre-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868884 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db992848-af0b-4a8d-aa98-711ff341ea78" containerName="payload-pre-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868890 4541 cpu_manager.go:414] "containerMap: removing container" podUID="7014bbd1-6465-4805-8c57-2bf4b6d98821" containerName="payload-pre-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868895 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7014bbd1-6465-4805-8c57-2bf4b6d98821" containerName="payload-pre-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868948 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="db992848-af0b-4a8d-aa98-711ff341ea78" containerName="payload-pre-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868955 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="7014bbd1-6465-4805-8c57-2bf4b6d98821" containerName="payload-pre-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868962 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8d1aab-adc2-4c58-9f30-da34e5d41754" containerName="revoke-keys" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868968 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="9303b207-db31-4672-aea3-cb93766856db" containerName="istio-proxy" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868973 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="1045d728-1c5d-49bf-9459-633d5d11aee3" containerName="maas-api" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868977 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74e0ba8-a137-4e34-9a7d-a7c4559fce1b" containerName="main" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868983 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e3e2e04-551b-4094-b4d4-6f35b41069ef" containerName="maas-api" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868990 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6edf898-5a79-4539-a61a-ea6a379c98d5" containerName="main" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.868995 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07c5e28-99e1-470a-a007-889ba3917657" containerName="istio-proxy" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.869001 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e55ba40-fb4f-4a14-8d11-8226803aa43c" containerName="istio-proxy" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.869007 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4e8e41-848f-4bb0-a195-d3a1282908b2" containerName="payload-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.869013 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="af19ec54-5e4a-402c-ad37-82f779d0cfef" containerName="payload-pre-processing" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.869018 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d2c9a3-a337-406d-a9ed-0d5d7e4bcfe0" containerName="main" Nov 03 14:32:49.869440 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.869023 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca463a2-02b8-46a2-8a66-fd398b50eafd" containerName="main" Nov 03 14:32:49.872032 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.872007 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:32:49.954075 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.954029 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/ecb6b791-4359-4837-ae32-66dd1e9ba99a-kube-api-access-jcdqv\") pod \"payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls\" (UID: \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\") " pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:32:49.954269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.954157 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb6b791-4359-4837-ae32-66dd1e9ba99a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls\" (UID: \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\") " pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:32:49.971886 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.971854 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f530cd7c-dockercfg-cwsrw\"" Nov 03 14:32:49.971886 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.971854 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f530cd7c\"" Nov 03 14:32:49.973662 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:49.973636 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls"] Nov 03 14:32:50.052866 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.052824 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps"] Nov 03 14:32:50.055497 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.055468 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/ecb6b791-4359-4837-ae32-66dd1e9ba99a-kube-api-access-jcdqv\") pod \"payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls\" (UID: \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\") " pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:32:50.055637 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.055520 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb6b791-4359-4837-ae32-66dd1e9ba99a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls\" (UID: \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\") " pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:32:50.056225 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.056205 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:32:50.056358 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.056211 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb6b791-4359-4837-ae32-66dd1e9ba99a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls\" (UID: \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\") " pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:32:50.111534 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.111499 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps"] Nov 03 14:32:50.113666 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.113635 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/ecb6b791-4359-4837-ae32-66dd1e9ba99a-kube-api-access-jcdqv\") pod \"payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls\" (UID: \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\") " pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:32:50.156635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.156605 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98jz\" (UniqueName: \"kubernetes.io/projected/e55ba705-fec0-42af-b2ef-cef34b3a65a4-kube-api-access-t98jz\") pod \"payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps\" (UID: \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:32:50.156864 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.156721 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55ba705-fec0-42af-b2ef-cef34b3a65a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps\" (UID: \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:32:50.185038 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.185006 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:32:50.257734 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.257695 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98jz\" (UniqueName: \"kubernetes.io/projected/e55ba705-fec0-42af-b2ef-cef34b3a65a4-kube-api-access-t98jz\") pod \"payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps\" (UID: \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:32:50.257909 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.257788 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55ba705-fec0-42af-b2ef-cef34b3a65a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps\" (UID: \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:32:50.258583 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.258563 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55ba705-fec0-42af-b2ef-cef34b3a65a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps\" (UID: \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:32:50.312187 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.312153 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98jz\" (UniqueName: \"kubernetes.io/projected/e55ba705-fec0-42af-b2ef-cef34b3a65a4-kube-api-access-t98jz\") pod \"payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps\" (UID: \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:32:50.368822 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.368764 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:32:50.558417 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.558393 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls"] Nov 03 14:32:50.560976 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:32:50.560941 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb6b791_4359_4837_ae32_66dd1e9ba99a.slice/crio-513dde0422781baeb12d326fe33ff24fe7314f3ac5ed6059a219ae27219562c7 WatchSource:0}: Error finding container 513dde0422781baeb12d326fe33ff24fe7314f3ac5ed6059a219ae27219562c7: Status 404 returned error can't find the container with id 513dde0422781baeb12d326fe33ff24fe7314f3ac5ed6059a219ae27219562c7 Nov 03 14:32:50.754094 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.754059 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps"] Nov 03 14:32:50.754954 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:32:50.754923 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55ba705_fec0_42af_b2ef_cef34b3a65a4.slice/crio-1dfcb367fc3516d988aef1a7f7018a1f24a08c1c5aba912b285a10a0064594dd WatchSource:0}: Error finding container 1dfcb367fc3516d988aef1a7f7018a1f24a08c1c5aba912b285a10a0064594dd: Status 404 returned error can't find the container with id 1dfcb367fc3516d988aef1a7f7018a1f24a08c1c5aba912b285a10a0064594dd Nov 03 14:32:50.955603 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.955551 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" event={"ID":"e55ba705-fec0-42af-b2ef-cef34b3a65a4","Type":"ContainerStarted","Data":"f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090"} Nov 03 14:32:50.956058 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.955618 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" event={"ID":"e55ba705-fec0-42af-b2ef-cef34b3a65a4","Type":"ContainerStarted","Data":"1dfcb367fc3516d988aef1a7f7018a1f24a08c1c5aba912b285a10a0064594dd"} Nov 03 14:32:50.956058 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.955741 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:32:50.957093 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.957067 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" event={"ID":"ecb6b791-4359-4837-ae32-66dd1e9ba99a","Type":"ContainerStarted","Data":"24a7b3b9b9acfcef8c3a23b61f2cb7a5a6c4b06d21038400fba32f4e369dc293"} Nov 03 14:32:50.957226 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.957108 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" event={"ID":"ecb6b791-4359-4837-ae32-66dd1e9ba99a","Type":"ContainerStarted","Data":"513dde0422781baeb12d326fe33ff24fe7314f3ac5ed6059a219ae27219562c7"} Nov 03 14:32:50.957226 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:50.957198 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:32:51.065446 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:51.065338 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" podStartSLOduration=1.06532542 podStartE2EDuration="1.06532542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:51.064576192 +0000 UTC m=+5302.378548053" watchObservedRunningTime="2026-11-03 14:32:51.06532542 +0000 UTC m=+5302.379297299" Nov 03 14:32:51.177007 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:32:51.176949 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" podStartSLOduration=2.176935991 podStartE2EDuration="2.176935991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:51.176407488 +0000 UTC m=+5302.490379349" watchObservedRunningTime="2026-11-03 14:32:51.176935991 +0000 UTC m=+5302.490907851" Nov 03 14:33:01.964499 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:01.964470 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:33:01.964951 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:01.964542 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:33:16.888099 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:33:16.888062 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:33:33.997818 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:33.997765 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls"] Nov 03 14:33:34.106786 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:34.106748 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps"] Nov 03 14:33:38.998947 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:38.998886 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" podUID="ecb6b791-4359-4837-ae32-66dd1e9ba99a" containerName="payload-processing" containerID="cri-o://24a7b3b9b9acfcef8c3a23b61f2cb7a5a6c4b06d21038400fba32f4e369dc293" gracePeriod=25 Nov 03 14:33:39.107723 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.107686 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" podUID="e55ba705-fec0-42af-b2ef-cef34b3a65a4" containerName="payload-pre-processing" containerID="cri-o://f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090" gracePeriod=25 Nov 03 14:33:39.165819 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.165756 4541 generic.go:356] "Generic (PLEG): container finished" podID="ecb6b791-4359-4837-ae32-66dd1e9ba99a" containerID="24a7b3b9b9acfcef8c3a23b61f2cb7a5a6c4b06d21038400fba32f4e369dc293" exitCode=0 Nov 03 14:33:39.165819 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.165809 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" event={"ID":"ecb6b791-4359-4837-ae32-66dd1e9ba99a","Type":"ContainerDied","Data":"24a7b3b9b9acfcef8c3a23b61f2cb7a5a6c4b06d21038400fba32f4e369dc293"} Nov 03 14:33:39.252179 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.252107 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:33:39.350394 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.350368 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:33:39.387501 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.387475 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecb6b791-4359-4837-ae32-66dd1e9ba99a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb6b791-4359-4837-ae32-66dd1e9ba99a-plugins-config-volume\") pod \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\" (UID: \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\") " Nov 03 14:33:39.387635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.387558 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecb6b791-4359-4837-ae32-66dd1e9ba99a-kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/ecb6b791-4359-4837-ae32-66dd1e9ba99a-kube-api-access-jcdqv\") pod \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\" (UID: \"ecb6b791-4359-4837-ae32-66dd1e9ba99a\") " Nov 03 14:33:39.387873 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.387853 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb6b791-4359-4837-ae32-66dd1e9ba99a-plugins-config-volume" pod "ecb6b791-4359-4837-ae32-66dd1e9ba99a" (UID: "ecb6b791-4359-4837-ae32-66dd1e9ba99a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:39.389610 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.389579 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb6b791-4359-4837-ae32-66dd1e9ba99a-kube-api-access-jcdqv" pod "ecb6b791-4359-4837-ae32-66dd1e9ba99a" (UID: "ecb6b791-4359-4837-ae32-66dd1e9ba99a"). InnerVolumeSpecName "kube-api-access-jcdqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:39.488819 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.488788 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e55ba705-fec0-42af-b2ef-cef34b3a65a4-kube-api-access-t98jz\" (UniqueName: \"kubernetes.io/projected/e55ba705-fec0-42af-b2ef-cef34b3a65a4-kube-api-access-t98jz\") pod \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\" (UID: \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\") " Nov 03 14:33:39.488985 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.488881 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e55ba705-fec0-42af-b2ef-cef34b3a65a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55ba705-fec0-42af-b2ef-cef34b3a65a4-plugins-config-volume\") pod \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\" (UID: \"e55ba705-fec0-42af-b2ef-cef34b3a65a4\") " Nov 03 14:33:39.489102 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.489087 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcdqv\" (UniqueName: \"kubernetes.io/projected/ecb6b791-4359-4837-ae32-66dd1e9ba99a-kube-api-access-jcdqv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:39.489169 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.489103 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecb6b791-4359-4837-ae32-66dd1e9ba99a-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:39.489241 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.489216 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e55ba705-fec0-42af-b2ef-cef34b3a65a4-plugins-config-volume" pod "e55ba705-fec0-42af-b2ef-cef34b3a65a4" (UID: "e55ba705-fec0-42af-b2ef-cef34b3a65a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:39.490923 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.490900 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55ba705-fec0-42af-b2ef-cef34b3a65a4-kube-api-access-t98jz" pod "e55ba705-fec0-42af-b2ef-cef34b3a65a4" (UID: "e55ba705-fec0-42af-b2ef-cef34b3a65a4"). InnerVolumeSpecName "kube-api-access-t98jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:39.590446 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.590371 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e55ba705-fec0-42af-b2ef-cef34b3a65a4-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:39.590446 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:39.590400 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t98jz\" (UniqueName: \"kubernetes.io/projected/e55ba705-fec0-42af-b2ef-cef34b3a65a4-kube-api-access-t98jz\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:40.171850 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.171815 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" Nov 03 14:33:40.172339 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.171808 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls" event={"ID":"ecb6b791-4359-4837-ae32-66dd1e9ba99a","Type":"ContainerDied","Data":"513dde0422781baeb12d326fe33ff24fe7314f3ac5ed6059a219ae27219562c7"} Nov 03 14:33:40.172339 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.172001 4541 scope.go:122] "RemoveContainer" containerID="24a7b3b9b9acfcef8c3a23b61f2cb7a5a6c4b06d21038400fba32f4e369dc293" Nov 03 14:33:40.173307 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.173199 4541 generic.go:356] "Generic (PLEG): container finished" podID="e55ba705-fec0-42af-b2ef-cef34b3a65a4" containerID="f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090" exitCode=0 Nov 03 14:33:40.173307 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.173273 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" Nov 03 14:33:40.173400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.173273 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" event={"ID":"e55ba705-fec0-42af-b2ef-cef34b3a65a4","Type":"ContainerDied","Data":"f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090"} Nov 03 14:33:40.173400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.173387 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps" event={"ID":"e55ba705-fec0-42af-b2ef-cef34b3a65a4","Type":"ContainerDied","Data":"1dfcb367fc3516d988aef1a7f7018a1f24a08c1c5aba912b285a10a0064594dd"} Nov 03 14:33:40.181994 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.181978 4541 scope.go:122] "RemoveContainer" containerID="f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090" Nov 03 14:33:40.189707 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.189690 4541 scope.go:122] "RemoveContainer" containerID="f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090" Nov 03 14:33:40.189991 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:33:40.189970 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090\": container with ID starting with f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090 not found: ID does not exist" containerID="f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090" Nov 03 14:33:40.190032 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.189998 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090"} err="failed to get container status \"f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090\": rpc error: code = NotFound desc = could not find container \"f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090\": container with ID starting with f662e536530ed8177981edda56993189ee29867d5c16600f77a6acd5a0e5a090 not found: ID does not exist" Nov 03 14:33:40.327972 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.327889 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps"] Nov 03 14:33:40.331728 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.331703 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f530cd7c-69fd7b9779-7dqps"] Nov 03 14:33:40.435046 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.435014 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls"] Nov 03 14:33:40.438574 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.438548 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f530cd7c-6dbccd5f8f-xlfls"] Nov 03 14:33:40.866617 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.866582 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e55ba705-fec0-42af-b2ef-cef34b3a65a4" path="/var/lib/kubelet/pods/e55ba705-fec0-42af-b2ef-cef34b3a65a4/volumes" Nov 03 14:33:40.866974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:33:40.866960 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecb6b791-4359-4837-ae32-66dd1e9ba99a" path="/var/lib/kubelet/pods/ecb6b791-4359-4837-ae32-66dd1e9ba99a/volumes" Nov 03 14:33:46.889177 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:33:46.889138 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:34:16.892008 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:34:16.891970 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:34:29.534737 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:29.534708 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:34:29.546140 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:29.546113 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:34:45.891278 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:45.891241 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x"] Nov 03 14:34:45.891692 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:45.891577 4541 cpu_manager.go:414] "containerMap: removing container" podUID="ecb6b791-4359-4837-ae32-66dd1e9ba99a" containerName="payload-processing" Nov 03 14:34:45.891692 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:45.891589 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecb6b791-4359-4837-ae32-66dd1e9ba99a" containerName="payload-processing" Nov 03 14:34:45.891692 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:45.891600 4541 cpu_manager.go:414] "containerMap: removing container" podUID="e55ba705-fec0-42af-b2ef-cef34b3a65a4" containerName="payload-pre-processing" Nov 03 14:34:45.891692 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:45.891606 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e55ba705-fec0-42af-b2ef-cef34b3a65a4" containerName="payload-pre-processing" Nov 03 14:34:45.891692 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:45.891677 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="e55ba705-fec0-42af-b2ef-cef34b3a65a4" containerName="payload-pre-processing" Nov 03 14:34:45.891692 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:45.891694 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecb6b791-4359-4837-ae32-66dd1e9ba99a" containerName="payload-processing" Nov 03 14:34:45.894538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:45.894513 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.002100 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.001998 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x"] Nov 03 14:34:46.074750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.074715 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.074750 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.074748 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7828782e-ff3e-4876-b530-af687a56b074-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.074959 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.074791 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.074959 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.074879 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.074959 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.074915 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7828782e-ff3e-4876-b530-af687a56b074-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.075088 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.074992 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqvh\" (UniqueName: \"kubernetes.io/projected/7828782e-ff3e-4876-b530-af687a56b074-kube-api-access-sdqvh\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.075088 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.075039 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.075088 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.075069 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7828782e-ff3e-4876-b530-af687a56b074-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.075192 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.075095 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176120 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176087 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176136 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7828782e-ff3e-4876-b530-af687a56b074-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176172 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176203 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176225 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7828782e-ff3e-4876-b530-af687a56b074-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176257 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176308 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176281 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176338 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7828782e-ff3e-4876-b530-af687a56b074-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176445 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqvh\" (UniqueName: \"kubernetes.io/projected/7828782e-ff3e-4876-b530-af687a56b074-kube-api-access-sdqvh\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176667 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176604 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176918 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176891 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.176918 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176907 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.177040 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.176967 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.177040 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.177030 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7828782e-ff3e-4876-b530-af687a56b074-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.178594 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.178565 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7828782e-ff3e-4876-b530-af687a56b074-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.178816 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.178798 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7828782e-ff3e-4876-b530-af687a56b074-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.231884 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.231856 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7828782e-ff3e-4876-b530-af687a56b074-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.232013 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.231987 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqvh\" (UniqueName: \"kubernetes.io/projected/7828782e-ff3e-4876-b530-af687a56b074-kube-api-access-sdqvh\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jkh4x\" (UID: \"7828782e-ff3e-4876-b530-af687a56b074\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.507338 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.507256 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:46.889261 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:34:46.889188 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:34:46.924057 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.924033 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x"] Nov 03 14:34:46.926224 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:34:46.926190 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7828782e_ff3e_4876_b530_af687a56b074.slice/crio-9a52d8312a87b910c08d21a95737a2c4e962502c382a3c9cdeca46ead75f1ad4 WatchSource:0}: Error finding container 9a52d8312a87b910c08d21a95737a2c4e962502c382a3c9cdeca46ead75f1ad4: Status 404 returned error can't find the container with id 9a52d8312a87b910c08d21a95737a2c4e962502c382a3c9cdeca46ead75f1ad4 Nov 03 14:34:46.928257 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.928225 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:34:46.928343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.928286 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:34:46.928343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:46.928316 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:34:47.472500 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:47.472462 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" event={"ID":"7828782e-ff3e-4876-b530-af687a56b074","Type":"ContainerStarted","Data":"6cd5da1ee9efd9a30c48c0ff64a54887c450be4116d055c0c2935a2c84c1e588"} Nov 03 14:34:47.472728 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:47.472529 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" event={"ID":"7828782e-ff3e-4876-b530-af687a56b074","Type":"ContainerStarted","Data":"9a52d8312a87b910c08d21a95737a2c4e962502c382a3c9cdeca46ead75f1ad4"} Nov 03 14:34:47.507830 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:47.507801 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:47.509412 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:47.509385 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 14:34:47.509556 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:47.509452 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 14:34:48.507578 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:48.507537 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 14:34:48.508116 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:48.507599 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 14:34:49.507969 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:49.507931 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 14:34:49.508390 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:49.507988 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 14:34:50.508189 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:50.508154 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 14:34:50.508631 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:50.508217 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 14:34:51.508480 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:51.508441 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 14:34:51.508929 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:51.508511 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 14:34:52.508474 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:52.508429 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 14:34:52.508630 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:52.508507 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 14:34:53.508504 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:53.508466 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 14:34:53.508685 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:53.508519 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 14:34:54.516867 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:54.516824 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 03 14:34:54.517364 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:54.516891 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 03 14:34:56.548923 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:56.548885 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:34:56.549466 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:56.548954 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:34:57.407179 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:57.407146 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:57.407546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:57.407523 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:57.408578 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:57.408558 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" Nov 03 14:34:57.569734 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:34:57.569678 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podStartSLOduration=12.569664656 podStartE2EDuration="12.569664656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:47.667659648 +0000 UTC m=+5418.981631510" watchObservedRunningTime="2026-11-03 14:34:57.569664656 +0000 UTC m=+5428.883636515" Nov 03 14:35:13.052359 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:13.052257 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk"] Nov 03 14:35:13.099628 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:13.099595 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de9cea6c-vv5nk"] Nov 03 14:35:14.868627 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:14.868592 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8d1aab-adc2-4c58-9f30-da34e5d41754" path="/var/lib/kubelet/pods/3e8d1aab-adc2-4c58-9f30-da34e5d41754/volumes" Nov 03 14:35:16.166062 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.166030 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d"] Nov 03 14:35:16.170675 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.170653 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.253714 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.253678 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.254035 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.254008 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.254212 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.254192 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.254341 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.254323 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06cc0694-c336-47b3-b0c8-856a3c2441c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.254461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.254444 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hzv\" (UniqueName: \"kubernetes.io/projected/06cc0694-c336-47b3-b0c8-856a3c2441c0-kube-api-access-s7hzv\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.254614 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.254597 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.254744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.254727 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.255619 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.255596 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.255832 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.255810 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.357326 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.357291 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.357538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.357346 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.357538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.357366 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.357538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.357393 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.357538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.357411 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06cc0694-c336-47b3-b0c8-856a3c2441c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.357538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.357427 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hzv\" (UniqueName: \"kubernetes.io/projected/06cc0694-c336-47b3-b0c8-856a3c2441c0-kube-api-access-s7hzv\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.357538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.357460 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.357538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.357482 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.357538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.357508 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.358304 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.358279 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.358646 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.358618 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.358971 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.358932 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.359540 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.359492 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.359540 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.359527 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06cc0694-c336-47b3-b0c8-856a3c2441c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.369023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.368999 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.371695 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.371644 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.384939 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.379258 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d"] Nov 03 14:35:16.423494 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.423387 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06cc0694-c336-47b3-b0c8-856a3c2441c0-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.424335 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.424299 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hzv\" (UniqueName: \"kubernetes.io/projected/06cc0694-c336-47b3-b0c8-856a3c2441c0-kube-api-access-s7hzv\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-68h6d\" (UID: \"06cc0694-c336-47b3-b0c8-856a3c2441c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.501073 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.501038 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:16.889095 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:35:16.889015 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:35:16.942522 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.942488 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d"] Nov 03 14:35:16.945670 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:35:16.945638 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cc0694_c336_47b3_b0c8_856a3c2441c0.slice/crio-43e6cdd6e3c71f959512cd9a95ea046444a7151428d90edea5f81cf1edd7d920 WatchSource:0}: Error finding container 43e6cdd6e3c71f959512cd9a95ea046444a7151428d90edea5f81cf1edd7d920: Status 404 returned error can't find the container with id 43e6cdd6e3c71f959512cd9a95ea046444a7151428d90edea5f81cf1edd7d920 Nov 03 14:35:16.947921 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.947881 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:35:16.948007 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.947971 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:35:16.948056 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:16.948039 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:35:17.612096 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:17.612056 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" event={"ID":"06cc0694-c336-47b3-b0c8-856a3c2441c0","Type":"ContainerStarted","Data":"34844bbb2074fe7d1fb65f956134e1e2ba271248e4bd8ca08be6c43b3e1ce2e4"} Nov 03 14:35:17.612555 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:17.612112 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" event={"ID":"06cc0694-c336-47b3-b0c8-856a3c2441c0","Type":"ContainerStarted","Data":"43e6cdd6e3c71f959512cd9a95ea046444a7151428d90edea5f81cf1edd7d920"} Nov 03 14:35:17.777235 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:17.777171 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podStartSLOduration=1.7771545 podStartE2EDuration="1.7771545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:17.77607189 +0000 UTC m=+5449.090043752" watchObservedRunningTime="2026-11-03 14:35:17.7771545 +0000 UTC m=+5449.091126356" Nov 03 14:35:18.502284 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:18.502241 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:18.503925 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:18.503899 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 14:35:18.504047 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:18.503960 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 14:35:19.502029 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:19.501988 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 14:35:19.502442 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:19.502066 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 14:35:20.501520 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:20.501482 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 14:35:20.501703 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:20.501549 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 14:35:21.501854 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:21.501819 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 14:35:21.502288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:21.501886 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 14:35:22.502072 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:22.502036 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 14:35:22.502483 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:22.502102 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 14:35:23.502107 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:23.502058 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 14:35:23.502593 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:23.502131 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 14:35:24.502122 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:24.502087 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 03 14:35:24.502524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:24.502139 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 03 14:35:26.544935 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:26.544886 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:26.545622 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:26.545579 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:27.586958 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:27.586883 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:27.587488 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:27.586990 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:28.102331 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:28.102291 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:28.102737 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:28.102708 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:28.103542 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:35:28.103523 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" Nov 03 14:35:46.929352 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:35:46.929317 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:36:16.439250 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.439217 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt"] Nov 03 14:36:16.442917 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.442891 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.540423 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.540376 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-cb156fca-kserve-self-signed-certs\"" Nov 03 14:36:16.543818 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.543788 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt"] Nov 03 14:36:16.591166 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.591133 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b173d981-600e-4f5d-a8c7-8babec35d41f-tls-certs\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.591339 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.591187 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-tmp-dir\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.591339 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.591279 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-dshm\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.591339 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.591317 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/b173d981-600e-4f5d-a8c7-8babec35d41f-kube-api-access-zfzbk\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.591339 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.591334 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-model-cache\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.591482 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.591442 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-home\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.692139 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.692053 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-tmp-dir\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.692139 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.692105 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-dshm\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.692139 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.692133 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/b173d981-600e-4f5d-a8c7-8babec35d41f-kube-api-access-zfzbk\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.692403 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.692148 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-model-cache\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.692403 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.692170 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-home\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.692403 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.692376 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b173d981-600e-4f5d-a8c7-8babec35d41f-tls-certs\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.692568 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.692487 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-tmp-dir\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.692629 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.692576 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-home\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.692629 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.692600 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-model-cache\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.694445 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.694415 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-dshm\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.694745 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.694727 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b173d981-600e-4f5d-a8c7-8babec35d41f-tls-certs\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.747594 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.747555 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/b173d981-600e-4f5d-a8c7-8babec35d41f-kube-api-access-zfzbk\") pod \"e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.762672 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:16.762650 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:16.889196 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:36:16.889155 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:36:17.131839 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:17.131808 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt"] Nov 03 14:36:17.135031 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:36:17.135001 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb173d981_600e_4f5d_a8c7_8babec35d41f.slice/crio-63a5a2be950b2d80b501e7ee47810e5a6731b04d684cd16b1b2f5b0a342f9921 WatchSource:0}: Error finding container 63a5a2be950b2d80b501e7ee47810e5a6731b04d684cd16b1b2f5b0a342f9921: Status 404 returned error can't find the container with id 63a5a2be950b2d80b501e7ee47810e5a6731b04d684cd16b1b2f5b0a342f9921 Nov 03 14:36:17.939890 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:17.939850 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" event={"ID":"b173d981-600e-4f5d-a8c7-8babec35d41f","Type":"ContainerStarted","Data":"048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb"} Nov 03 14:36:17.940405 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:17.939905 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" event={"ID":"b173d981-600e-4f5d-a8c7-8babec35d41f","Type":"ContainerStarted","Data":"63a5a2be950b2d80b501e7ee47810e5a6731b04d684cd16b1b2f5b0a342f9921"} Nov 03 14:36:18.049416 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:18.049359 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" podStartSLOduration=2.04934589 podStartE2EDuration="2.04934589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:36:18.04835803 +0000 UTC m=+5509.362329891" watchObservedRunningTime="2026-11-03 14:36:18.04934589 +0000 UTC m=+5509.363317749" Nov 03 14:36:18.763700 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:18.763662 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:26.763446 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:26.763401 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:26.776425 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:26.776401 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:27.001476 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:27.001444 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:36:30.186194 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:36:30.186158 4541 scope.go:122] "RemoveContainer" containerID="6542a225014665d4b8445db376e9ab633fb1ae79b75692321b7a792a907f6153" Nov 03 14:36:46.888639 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:36:46.888602 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:37:16.889367 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:37:16.889323 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:37:26.423234 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:26.423193 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt"] Nov 03 14:37:41.443289 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.443243 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" podUID="b173d981-600e-4f5d-a8c7-8babec35d41f" containerName="main" containerID="cri-o://048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb" gracePeriod=45 Nov 03 14:37:41.682831 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.682807 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:37:41.786400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786308 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-tmp-dir\") pod \"b173d981-600e-4f5d-a8c7-8babec35d41f\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " Nov 03 14:37:41.786400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786361 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b173d981-600e-4f5d-a8c7-8babec35d41f-tls-certs\" (UniqueName: \"kubernetes.io/secret/b173d981-600e-4f5d-a8c7-8babec35d41f-tls-certs\") pod \"b173d981-600e-4f5d-a8c7-8babec35d41f\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " Nov 03 14:37:41.786400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786401 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b173d981-600e-4f5d-a8c7-8babec35d41f-kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/b173d981-600e-4f5d-a8c7-8babec35d41f-kube-api-access-zfzbk\") pod \"b173d981-600e-4f5d-a8c7-8babec35d41f\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " Nov 03 14:37:41.786697 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786419 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-model-cache\") pod \"b173d981-600e-4f5d-a8c7-8babec35d41f\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " Nov 03 14:37:41.786697 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786470 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-home\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-home\") pod \"b173d981-600e-4f5d-a8c7-8babec35d41f\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " Nov 03 14:37:41.786697 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786503 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-dshm\") pod \"b173d981-600e-4f5d-a8c7-8babec35d41f\" (UID: \"b173d981-600e-4f5d-a8c7-8babec35d41f\") " Nov 03 14:37:41.786697 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786618 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-tmp-dir" pod "b173d981-600e-4f5d-a8c7-8babec35d41f" (UID: "b173d981-600e-4f5d-a8c7-8babec35d41f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:41.786974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786733 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-model-cache" pod "b173d981-600e-4f5d-a8c7-8babec35d41f" (UID: "b173d981-600e-4f5d-a8c7-8babec35d41f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:41.786974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786750 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:41.786974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.786802 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-home" pod "b173d981-600e-4f5d-a8c7-8babec35d41f" (UID: "b173d981-600e-4f5d-a8c7-8babec35d41f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:41.788793 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.788755 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b173d981-600e-4f5d-a8c7-8babec35d41f-tls-certs" pod "b173d981-600e-4f5d-a8c7-8babec35d41f" (UID: "b173d981-600e-4f5d-a8c7-8babec35d41f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:37:41.788905 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.788789 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-dshm" pod "b173d981-600e-4f5d-a8c7-8babec35d41f" (UID: "b173d981-600e-4f5d-a8c7-8babec35d41f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:37:41.788905 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.788859 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b173d981-600e-4f5d-a8c7-8babec35d41f-kube-api-access-zfzbk" pod "b173d981-600e-4f5d-a8c7-8babec35d41f" (UID: "b173d981-600e-4f5d-a8c7-8babec35d41f"). InnerVolumeSpecName "kube-api-access-zfzbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:37:41.887224 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.887189 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:41.887224 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.887217 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b173d981-600e-4f5d-a8c7-8babec35d41f-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:41.887224 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.887227 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfzbk\" (UniqueName: \"kubernetes.io/projected/b173d981-600e-4f5d-a8c7-8babec35d41f-kube-api-access-zfzbk\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:41.887470 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.887237 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:41.887470 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:41.887246 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b173d981-600e-4f5d-a8c7-8babec35d41f-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:37:42.392879 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.392832 4541 generic.go:356] "Generic (PLEG): container finished" podID="b173d981-600e-4f5d-a8c7-8babec35d41f" containerID="048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb" exitCode=0 Nov 03 14:37:42.392879 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.392883 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" Nov 03 14:37:42.393179 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.392876 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" event={"ID":"b173d981-600e-4f5d-a8c7-8babec35d41f","Type":"ContainerDied","Data":"048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb"} Nov 03 14:37:42.393179 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.392999 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt" event={"ID":"b173d981-600e-4f5d-a8c7-8babec35d41f","Type":"ContainerDied","Data":"63a5a2be950b2d80b501e7ee47810e5a6731b04d684cd16b1b2f5b0a342f9921"} Nov 03 14:37:42.393179 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.393024 4541 scope.go:122] "RemoveContainer" containerID="048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb" Nov 03 14:37:42.401011 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:37:42.400979 4541 helpers.go:246] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb173d981_600e_4f5d_a8c7_8babec35d41f.slice/pids.max": read /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb173d981_600e_4f5d_a8c7_8babec35d41f.slice/pids.max: no such device Nov 03 14:37:42.401969 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.401948 4541 scope.go:122] "RemoveContainer" containerID="048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb" Nov 03 14:37:42.402264 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:37:42.402243 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb\": container with ID starting with 048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb not found: ID does not exist" containerID="048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb" Nov 03 14:37:42.402316 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.402272 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb"} err="failed to get container status \"048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb\": rpc error: code = NotFound desc = could not find container \"048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb\": container with ID starting with 048bd369e3d42f3e97896dfe6785ab873a9decb6e3619dc6dcbc2076874dd7cb not found: ID does not exist" Nov 03 14:37:42.548052 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.548014 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt"] Nov 03 14:37:42.595152 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.595111 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-69f98b56b4-jptlt"] Nov 03 14:37:42.866607 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:42.866571 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b173d981-600e-4f5d-a8c7-8babec35d41f" path="/var/lib/kubelet/pods/b173d981-600e-4f5d-a8c7-8babec35d41f/volumes" Nov 03 14:37:45.188597 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.188560 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq"] Nov 03 14:37:45.189011 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.188938 4541 cpu_manager.go:414] "containerMap: removing container" podUID="b173d981-600e-4f5d-a8c7-8babec35d41f" containerName="main" Nov 03 14:37:45.189011 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.188950 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b173d981-600e-4f5d-a8c7-8babec35d41f" containerName="main" Nov 03 14:37:45.189087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.189046 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="b173d981-600e-4f5d-a8c7-8babec35d41f" containerName="main" Nov 03 14:37:45.193614 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.193592 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.248207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.248173 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-8366e3-openshift-default-dockercfg-nghn4\"" Nov 03 14:37:45.297689 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.297658 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq"] Nov 03 14:37:45.314074 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.314036 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-certs\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.314353 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.314097 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g6vv\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-kube-api-access-4g6vv\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.314353 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.314257 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-data\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.314353 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.314324 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-socket\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.314527 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.314489 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-envoy\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.314622 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.314604 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-podinfo\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.314673 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.314641 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-token\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.314758 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.314737 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istiod-ca-cert\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.314897 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.314798 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-credential-socket\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416009 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.415971 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-envoy\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416214 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416048 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-podinfo\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416214 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416074 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-token\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416214 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416107 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istiod-ca-cert\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416214 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416134 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-credential-socket\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416486 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416322 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-certs\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416486 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416410 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g6vv\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-kube-api-access-4g6vv\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416486 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416462 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-data\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416632 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416492 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-socket\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416632 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416514 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-credential-socket\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416755 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416727 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-certs\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.416944 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416921 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-socket\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.417008 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416938 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-data\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.417008 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.416931 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istiod-ca-cert\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.418416 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.418367 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-envoy\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.418534 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.418517 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-podinfo\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.472791 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.472709 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g6vv\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-kube-api-access-4g6vv\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.472791 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.472708 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-token\") pod \"e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.505928 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.505886 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:45.922543 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.922512 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq"] Nov 03 14:37:45.924440 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:37:45.924409 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2616fc02_53c8_4676_a91a_0ba4e99e9c00.slice/crio-ff53a516c35276423279b63fa16fe8fe2870b555fb40f3c274cb1085b661faf1 WatchSource:0}: Error finding container ff53a516c35276423279b63fa16fe8fe2870b555fb40f3c274cb1085b661faf1: Status 404 returned error can't find the container with id ff53a516c35276423279b63fa16fe8fe2870b555fb40f3c274cb1085b661faf1 Nov 03 14:37:45.926551 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.926518 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:37:45.926603 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.926582 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:37:45.926641 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:45.926613 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:37:46.412797 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:46.412742 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" event={"ID":"2616fc02-53c8-4676-a91a-0ba4e99e9c00","Type":"ContainerStarted","Data":"d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2"} Nov 03 14:37:46.413201 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:46.412822 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" event={"ID":"2616fc02-53c8-4676-a91a-0ba4e99e9c00","Type":"ContainerStarted","Data":"ff53a516c35276423279b63fa16fe8fe2870b555fb40f3c274cb1085b661faf1"} Nov 03 14:37:46.506837 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:46.506801 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:46.512028 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:46.512002 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:46.525847 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:46.525800 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" podStartSLOduration=1.525766522 podStartE2EDuration="1.525766522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:46.524728819 +0000 UTC m=+5597.838700707" watchObservedRunningTime="2026-11-03 14:37:46.525766522 +0000 UTC m=+5597.839738382" Nov 03 14:37:46.888939 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:37:46.888858 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:37:47.417945 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:47.417916 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:47.418731 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:47.418710 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:37:53.347941 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.347906 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8"] Nov 03 14:37:53.351967 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.351940 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:37:53.450516 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.450487 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8366e3\"" Nov 03 14:37:53.450680 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.450593 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bf17b8fc-dockercfg-5xxgz\"" Nov 03 14:37:53.452321 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.452301 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8"] Nov 03 14:37:53.519220 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.519185 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43d712c-24d9-4137-9a3a-fb3052623a07-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8\" (UID: \"d43d712c-24d9-4137-9a3a-fb3052623a07\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:37:53.519405 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.519325 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl57q\" (UniqueName: \"kubernetes.io/projected/d43d712c-24d9-4137-9a3a-fb3052623a07-kube-api-access-bl57q\") pod \"payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8\" (UID: \"d43d712c-24d9-4137-9a3a-fb3052623a07\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:37:53.531348 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.531316 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm"] Nov 03 14:37:53.534690 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.534672 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:37:53.589280 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.589249 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm"] Nov 03 14:37:53.620289 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.620214 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43d712c-24d9-4137-9a3a-fb3052623a07-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8\" (UID: \"d43d712c-24d9-4137-9a3a-fb3052623a07\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:37:53.620444 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.620314 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c42b708-0024-4dc8-855a-f2d53d6c33f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm\" (UID: \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:37:53.620444 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.620352 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4j7q\" (UniqueName: \"kubernetes.io/projected/1c42b708-0024-4dc8-855a-f2d53d6c33f6-kube-api-access-d4j7q\") pod \"payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm\" (UID: \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:37:53.620444 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.620379 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl57q\" (UniqueName: \"kubernetes.io/projected/d43d712c-24d9-4137-9a3a-fb3052623a07-kube-api-access-bl57q\") pod \"payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8\" (UID: \"d43d712c-24d9-4137-9a3a-fb3052623a07\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:37:53.620993 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.620970 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43d712c-24d9-4137-9a3a-fb3052623a07-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8\" (UID: \"d43d712c-24d9-4137-9a3a-fb3052623a07\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:37:53.674380 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.674347 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl57q\" (UniqueName: \"kubernetes.io/projected/d43d712c-24d9-4137-9a3a-fb3052623a07-kube-api-access-bl57q\") pod \"payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8\" (UID: \"d43d712c-24d9-4137-9a3a-fb3052623a07\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:37:53.721078 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.721044 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c42b708-0024-4dc8-855a-f2d53d6c33f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm\" (UID: \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:37:53.721236 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.721089 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4j7q\" (UniqueName: \"kubernetes.io/projected/1c42b708-0024-4dc8-855a-f2d53d6c33f6-kube-api-access-d4j7q\") pod \"payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm\" (UID: \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:37:53.721739 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.721716 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c42b708-0024-4dc8-855a-f2d53d6c33f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm\" (UID: \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:37:53.774795 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.774738 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4j7q\" (UniqueName: \"kubernetes.io/projected/1c42b708-0024-4dc8-855a-f2d53d6c33f6-kube-api-access-d4j7q\") pod \"payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm\" (UID: \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:37:53.846969 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.846918 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:37:53.969571 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:53.969536 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:37:54.212644 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.212619 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm"] Nov 03 14:37:54.214319 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:37:54.214282 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c42b708_0024_4dc8_855a_f2d53d6c33f6.slice/crio-d025d0da8afe5e39d4d9086c24cbaa2cb2d327119f87d4e3e954f230ed913852 WatchSource:0}: Error finding container d025d0da8afe5e39d4d9086c24cbaa2cb2d327119f87d4e3e954f230ed913852: Status 404 returned error can't find the container with id d025d0da8afe5e39d4d9086c24cbaa2cb2d327119f87d4e3e954f230ed913852 Nov 03 14:37:54.330489 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.330459 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8"] Nov 03 14:37:54.332794 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:37:54.332749 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43d712c_24d9_4137_9a3a_fb3052623a07.slice/crio-2770292d02a819a3603ef0d69dddd4b409323d93ab7d88ee0804ff745eb22a1d WatchSource:0}: Error finding container 2770292d02a819a3603ef0d69dddd4b409323d93ab7d88ee0804ff745eb22a1d: Status 404 returned error can't find the container with id 2770292d02a819a3603ef0d69dddd4b409323d93ab7d88ee0804ff745eb22a1d Nov 03 14:37:54.448431 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.448399 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" event={"ID":"1c42b708-0024-4dc8-855a-f2d53d6c33f6","Type":"ContainerStarted","Data":"671925d7b8dcc3c67bfb47dfe3b11c623539245980c1a7479866cd47c56a358b"} Nov 03 14:37:54.448913 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.448456 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" event={"ID":"1c42b708-0024-4dc8-855a-f2d53d6c33f6","Type":"ContainerStarted","Data":"d025d0da8afe5e39d4d9086c24cbaa2cb2d327119f87d4e3e954f230ed913852"} Nov 03 14:37:54.448913 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.448538 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:37:54.449885 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.449862 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" event={"ID":"d43d712c-24d9-4137-9a3a-fb3052623a07","Type":"ContainerStarted","Data":"c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9"} Nov 03 14:37:54.450001 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.449898 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" event={"ID":"d43d712c-24d9-4137-9a3a-fb3052623a07","Type":"ContainerStarted","Data":"2770292d02a819a3603ef0d69dddd4b409323d93ab7d88ee0804ff745eb22a1d"} Nov 03 14:37:54.450001 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.449965 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:37:54.556161 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.556055 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" podStartSLOduration=1.556043348 podStartE2EDuration="1.556043348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:54.555853338 +0000 UTC m=+5605.869825197" watchObservedRunningTime="2026-11-03 14:37:54.556043348 +0000 UTC m=+5605.870015202" Nov 03 14:37:54.663546 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:54.663494 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" podStartSLOduration=1.663479257 podStartE2EDuration="1.663479257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:54.663374047 +0000 UTC m=+5605.977345908" watchObservedRunningTime="2026-11-03 14:37:54.663479257 +0000 UTC m=+5605.977451116" Nov 03 14:37:56.415742 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.415707 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm"] Nov 03 14:37:56.419302 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.419280 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.471136 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.471102 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-8366e3-serving-cert\"" Nov 03 14:37:56.517669 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.517633 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm"] Nov 03 14:37:56.547715 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.547683 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.547893 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.547748 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.547938 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.547905 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9h5\" (UniqueName: \"kubernetes.io/projected/01791df9-2ba7-4cde-a425-7afbbacf95e2-kube-api-access-jf9h5\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.548049 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.548029 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01791df9-2ba7-4cde-a425-7afbbacf95e2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.649261 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.649211 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01791df9-2ba7-4cde-a425-7afbbacf95e2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.649484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.649301 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.649484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.649382 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.649484 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.649422 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9h5\" (UniqueName: \"kubernetes.io/projected/01791df9-2ba7-4cde-a425-7afbbacf95e2-kube-api-access-jf9h5\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.649484 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:37:56.649431 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-serving-cert: secret "maas-api-e2e-worker-single-main-8366e3-serving-cert" not found Nov 03 14:37:56.649709 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:37:56.649513 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls podName:01791df9-2ba7-4cde-a425-7afbbacf95e2 nodeName:}" failed. No retries permitted until 2026-11-03 14:37:57.149489406 +0000 UTC m=+5608.463461245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls") pod "maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" (UID: "01791df9-2ba7-4cde-a425-7afbbacf95e2") : secret "maas-api-e2e-worker-single-main-8366e3-serving-cert" not found Nov 03 14:37:56.650108 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.650081 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01791df9-2ba7-4cde-a425-7afbbacf95e2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.651986 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.651962 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:56.704467 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:56.704388 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9h5\" (UniqueName: \"kubernetes.io/projected/01791df9-2ba7-4cde-a425-7afbbacf95e2-kube-api-access-jf9h5\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:57.155010 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:57.154979 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:57.157514 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:57.157488 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:57.333445 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:57.333410 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:57.699113 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:57.699086 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm"] Nov 03 14:37:57.700453 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:37:57.700420 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01791df9_2ba7_4cde_a425_7afbbacf95e2.slice/crio-76d04c41c6c838a94fdfd4008f3ced62f6b98c8377914094164448502652bead WatchSource:0}: Error finding container 76d04c41c6c838a94fdfd4008f3ced62f6b98c8377914094164448502652bead: Status 404 returned error can't find the container with id 76d04c41c6c838a94fdfd4008f3ced62f6b98c8377914094164448502652bead Nov 03 14:37:57.701920 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:57.701904 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:37:58.468463 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:58.468427 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" event={"ID":"01791df9-2ba7-4cde-a425-7afbbacf95e2","Type":"ContainerStarted","Data":"76d04c41c6c838a94fdfd4008f3ced62f6b98c8377914094164448502652bead"} Nov 03 14:37:59.474118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:59.474077 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" event={"ID":"01791df9-2ba7-4cde-a425-7afbbacf95e2","Type":"ContainerStarted","Data":"c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469"} Nov 03 14:37:59.474526 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:59.474249 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:37:59.587635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:37:59.587583 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" podStartSLOduration=2.583818375 podStartE2EDuration="3.587570013s" totalImagesPullingTime="1.003751638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:37:59.585879767 +0000 UTC m=+5610.899851631" watchObservedRunningTime="2026-11-03 14:37:59.587570013 +0000 UTC m=+5610.901541893" Nov 03 14:38:05.461855 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:05.461824 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:38:05.462318 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:05.461903 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:38:05.483619 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:05.483591 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:38:14.543291 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:14.543255 4541 generic.go:356] "Generic (PLEG): container finished" podID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerID="d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2" exitCode=0 Nov 03 14:38:14.543903 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:14.543324 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" event={"ID":"2616fc02-53c8-4676-a91a-0ba4e99e9c00","Type":"ContainerDied","Data":"d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2"} Nov 03 14:38:14.543903 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:14.543647 4541 scope.go:122] "RemoveContainer" containerID="d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2" Nov 03 14:38:14.544919 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:14.544889 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:38:14.544989 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:14.544974 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:38:14.545029 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:14.545008 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:38:15.565586 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:15.565551 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" event={"ID":"2616fc02-53c8-4676-a91a-0ba4e99e9c00","Type":"ContainerStarted","Data":"79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980"} Nov 03 14:38:16.506599 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:16.506562 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:38:16.507948 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:16.507918 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:38:16.508080 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:16.507979 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 14:38:16.889411 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:38:16.889327 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:38:17.418509 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:17.418474 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:38:17.507058 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:17.507026 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:38:17.507240 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:17.507080 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 14:38:18.397040 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.397007 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv"] Nov 03 14:38:18.399468 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.399446 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.489174 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.489144 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-tmp-dir\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.489333 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.489208 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-dshm\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.489333 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.489306 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4226b\" (UniqueName: \"kubernetes.io/projected/0e448265-6edd-49ee-9fb1-02d58a2b7941-kube-api-access-4226b\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.489412 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.489349 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-home\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.489412 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.489368 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e448265-6edd-49ee-9fb1-02d58a2b7941-tls-certs\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.489511 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.489488 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-model-cache\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.504123 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.504097 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8366e3\"/\"openshift-service-ca.crt\"" Nov 03 14:38:18.504123 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.504114 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8366e3\"/\"kube-root-ca.crt\"" Nov 03 14:38:18.504286 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.504170 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8366e3\"/\"simulator-single-main-8366e3-kserve-self-signed-certs\"" Nov 03 14:38:18.504286 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.504202 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8366e3\"/\"default-dockercfg-nt7zl\"" Nov 03 14:38:18.506089 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.506072 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv"] Nov 03 14:38:18.506342 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.506324 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:38:18.506384 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.506365 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 14:38:18.590448 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.590414 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-model-cache\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.590631 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.590455 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-tmp-dir\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.590631 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.590522 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-dshm\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.590631 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.590577 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4226b\" (UniqueName: \"kubernetes.io/projected/0e448265-6edd-49ee-9fb1-02d58a2b7941-kube-api-access-4226b\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.590631 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.590610 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-home\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.590885 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.590635 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e448265-6edd-49ee-9fb1-02d58a2b7941-tls-certs\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.590947 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.590903 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-model-cache\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.590989 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.590970 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-tmp-dir\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.591027 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.590988 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-home\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.592981 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.592954 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-dshm\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.593151 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.593136 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e448265-6edd-49ee-9fb1-02d58a2b7941-tls-certs\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.660364 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.660325 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4226b\" (UniqueName: \"kubernetes.io/projected/0e448265-6edd-49ee-9fb1-02d58a2b7941-kube-api-access-4226b\") pod \"simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:18.710208 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:18.710176 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:19.098196 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:19.098166 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv"] Nov 03 14:38:19.100423 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:38:19.100391 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e448265_6edd_49ee_9fb1_02d58a2b7941.slice/crio-fcd15a8029c5b963a285bd2c1f8b3e9561d1972ddf9bedf6f646c0c4727067a5 WatchSource:0}: Error finding container fcd15a8029c5b963a285bd2c1f8b3e9561d1972ddf9bedf6f646c0c4727067a5: Status 404 returned error can't find the container with id fcd15a8029c5b963a285bd2c1f8b3e9561d1972ddf9bedf6f646c0c4727067a5 Nov 03 14:38:19.506467 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:19.506430 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:38:19.506903 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:19.506493 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 14:38:19.584594 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:19.584561 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" event={"ID":"0e448265-6edd-49ee-9fb1-02d58a2b7941","Type":"ContainerStarted","Data":"6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63"} Nov 03 14:38:19.584753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:19.584602 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" event={"ID":"0e448265-6edd-49ee-9fb1-02d58a2b7941","Type":"ContainerStarted","Data":"fcd15a8029c5b963a285bd2c1f8b3e9561d1972ddf9bedf6f646c0c4727067a5"} Nov 03 14:38:19.710767 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:19.710735 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:19.715087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:19.715034 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" podStartSLOduration=1.715018917 podStartE2EDuration="1.715018917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:38:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:38:19.714279063 +0000 UTC m=+5631.028250935" watchObservedRunningTime="2026-11-03 14:38:19.715018917 +0000 UTC m=+5631.028990780" Nov 03 14:38:20.507295 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:20.507258 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 03 14:38:20.507725 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:20.507312 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 03 14:38:21.511003 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:21.510968 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:38:21.511968 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:21.511944 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:38:28.710732 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:28.710685 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:28.723676 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:28.723648 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:29.642124 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:38:29.642091 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:38:46.888929 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:38:46.888886 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:39:16.889011 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:39:16.888972 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:39:19.359084 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:19.359051 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv"] Nov 03 14:39:21.676888 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:21.676863 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8"] Nov 03 14:39:21.796120 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:21.796086 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm"] Nov 03 14:39:26.677911 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:26.677851 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" podUID="d43d712c-24d9-4137-9a3a-fb3052623a07" containerName="payload-processing" containerID="cri-o://c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9" gracePeriod=25 Nov 03 14:39:26.797193 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:26.797152 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" podUID="1c42b708-0024-4dc8-855a-f2d53d6c33f6" containerName="payload-pre-processing" containerID="cri-o://671925d7b8dcc3c67bfb47dfe3b11c623539245980c1a7479866cd47c56a358b" gracePeriod=25 Nov 03 14:39:29.562332 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:29.562304 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:39:29.574209 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:29.574190 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:39:29.809979 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:29.809945 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm"] Nov 03 14:39:29.810171 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:29.810131 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" podUID="01791df9-2ba7-4cde-a425-7afbbacf95e2" containerName="maas-api" containerID="cri-o://c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469" gracePeriod=30 Nov 03 14:39:30.478929 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:30.478890 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" podUID="01791df9-2ba7-4cde-a425-7afbbacf95e2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.155:8443/health\": dial tcp 10.133.0.155:8443: connect: connection refused" Nov 03 14:39:31.058942 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.058917 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:39:31.081414 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.081392 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-metrics-certs\") pod \"01791df9-2ba7-4cde-a425-7afbbacf95e2\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " Nov 03 14:39:31.081495 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.081443 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls\") pod \"01791df9-2ba7-4cde-a425-7afbbacf95e2\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " Nov 03 14:39:31.081495 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.081463 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01791df9-2ba7-4cde-a425-7afbbacf95e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01791df9-2ba7-4cde-a425-7afbbacf95e2-openshift-service-ca\") pod \"01791df9-2ba7-4cde-a425-7afbbacf95e2\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " Nov 03 14:39:31.081586 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.081497 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01791df9-2ba7-4cde-a425-7afbbacf95e2-kube-api-access-jf9h5\" (UniqueName: \"kubernetes.io/projected/01791df9-2ba7-4cde-a425-7afbbacf95e2-kube-api-access-jf9h5\") pod \"01791df9-2ba7-4cde-a425-7afbbacf95e2\" (UID: \"01791df9-2ba7-4cde-a425-7afbbacf95e2\") " Nov 03 14:39:31.081895 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.081871 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01791df9-2ba7-4cde-a425-7afbbacf95e2-openshift-service-ca" pod "01791df9-2ba7-4cde-a425-7afbbacf95e2" (UID: "01791df9-2ba7-4cde-a425-7afbbacf95e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:31.083396 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.083372 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-metrics-certs" pod "01791df9-2ba7-4cde-a425-7afbbacf95e2" (UID: "01791df9-2ba7-4cde-a425-7afbbacf95e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:39:31.083467 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.083427 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01791df9-2ba7-4cde-a425-7afbbacf95e2-kube-api-access-jf9h5" pod "01791df9-2ba7-4cde-a425-7afbbacf95e2" (UID: "01791df9-2ba7-4cde-a425-7afbbacf95e2"). InnerVolumeSpecName "kube-api-access-jf9h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:31.083507 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.083483 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls" pod "01791df9-2ba7-4cde-a425-7afbbacf95e2" (UID: "01791df9-2ba7-4cde-a425-7afbbacf95e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:39:31.183057 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.183029 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:31.183175 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.183059 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01791df9-2ba7-4cde-a425-7afbbacf95e2-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:31.183175 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.183075 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01791df9-2ba7-4cde-a425-7afbbacf95e2-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:31.183175 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.183089 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf9h5\" (UniqueName: \"kubernetes.io/projected/01791df9-2ba7-4cde-a425-7afbbacf95e2-kube-api-access-jf9h5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:31.913403 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.913368 4541 generic.go:356] "Generic (PLEG): container finished" podID="01791df9-2ba7-4cde-a425-7afbbacf95e2" containerID="c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469" exitCode=0 Nov 03 14:39:31.913589 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.913421 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" Nov 03 14:39:31.913589 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.913442 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" event={"ID":"01791df9-2ba7-4cde-a425-7afbbacf95e2","Type":"ContainerDied","Data":"c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469"} Nov 03 14:39:31.913589 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.913482 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm" event={"ID":"01791df9-2ba7-4cde-a425-7afbbacf95e2","Type":"ContainerDied","Data":"76d04c41c6c838a94fdfd4008f3ced62f6b98c8377914094164448502652bead"} Nov 03 14:39:31.913589 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.913502 4541 scope.go:122] "RemoveContainer" containerID="c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469" Nov 03 14:39:31.923130 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.923107 4541 scope.go:122] "RemoveContainer" containerID="c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469" Nov 03 14:39:31.923384 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:39:31.923366 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469\": container with ID starting with c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469 not found: ID does not exist" containerID="c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469" Nov 03 14:39:31.923435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:31.923394 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469"} err="failed to get container status \"c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469\": rpc error: code = NotFound desc = could not find container \"c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469\": container with ID starting with c1bcadb1bdfa5d0e9d6e121556019120bec241ba9a27be51543d6efcb5b6d469 not found: ID does not exist" Nov 03 14:39:32.068194 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:32.068159 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm"] Nov 03 14:39:32.116459 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:32.116428 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-8366e3-84d9bdcc64-rm6mm"] Nov 03 14:39:32.867328 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:32.867292 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01791df9-2ba7-4cde-a425-7afbbacf95e2" path="/var/lib/kubelet/pods/01791df9-2ba7-4cde-a425-7afbbacf95e2/volumes" Nov 03 14:39:34.378923 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.378865 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" podUID="0e448265-6edd-49ee-9fb1-02d58a2b7941" containerName="main" containerID="cri-o://6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63" gracePeriod=45 Nov 03 14:39:34.641261 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.641199 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:39:34.715996 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.715948 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-tmp-dir\") pod \"0e448265-6edd-49ee-9fb1-02d58a2b7941\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " Nov 03 14:39:34.716199 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716026 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e448265-6edd-49ee-9fb1-02d58a2b7941-kube-api-access-4226b\" (UniqueName: \"kubernetes.io/projected/0e448265-6edd-49ee-9fb1-02d58a2b7941-kube-api-access-4226b\") pod \"0e448265-6edd-49ee-9fb1-02d58a2b7941\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " Nov 03 14:39:34.716199 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716067 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-model-cache\") pod \"0e448265-6edd-49ee-9fb1-02d58a2b7941\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " Nov 03 14:39:34.716199 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716087 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-dshm\") pod \"0e448265-6edd-49ee-9fb1-02d58a2b7941\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " Nov 03 14:39:34.716199 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716108 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-home\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-home\") pod \"0e448265-6edd-49ee-9fb1-02d58a2b7941\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " Nov 03 14:39:34.716199 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716139 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e448265-6edd-49ee-9fb1-02d58a2b7941-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e448265-6edd-49ee-9fb1-02d58a2b7941-tls-certs\") pod \"0e448265-6edd-49ee-9fb1-02d58a2b7941\" (UID: \"0e448265-6edd-49ee-9fb1-02d58a2b7941\") " Nov 03 14:39:34.716487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716240 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-tmp-dir" pod "0e448265-6edd-49ee-9fb1-02d58a2b7941" (UID: "0e448265-6edd-49ee-9fb1-02d58a2b7941"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:34.716487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716380 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-model-cache" pod "0e448265-6edd-49ee-9fb1-02d58a2b7941" (UID: "0e448265-6edd-49ee-9fb1-02d58a2b7941"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:34.716487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716459 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-home" pod "0e448265-6edd-49ee-9fb1-02d58a2b7941" (UID: "0e448265-6edd-49ee-9fb1-02d58a2b7941"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:34.716609 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716534 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:34.716609 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.716548 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:34.718433 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.718404 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e448265-6edd-49ee-9fb1-02d58a2b7941-tls-certs" pod "0e448265-6edd-49ee-9fb1-02d58a2b7941" (UID: "0e448265-6edd-49ee-9fb1-02d58a2b7941"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:39:34.718549 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.718431 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-dshm" pod "0e448265-6edd-49ee-9fb1-02d58a2b7941" (UID: "0e448265-6edd-49ee-9fb1-02d58a2b7941"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:34.718549 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.718464 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e448265-6edd-49ee-9fb1-02d58a2b7941-kube-api-access-4226b" pod "0e448265-6edd-49ee-9fb1-02d58a2b7941" (UID: "0e448265-6edd-49ee-9fb1-02d58a2b7941"). InnerVolumeSpecName "kube-api-access-4226b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:34.817268 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.817234 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4226b\" (UniqueName: \"kubernetes.io/projected/0e448265-6edd-49ee-9fb1-02d58a2b7941-kube-api-access-4226b\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:34.817268 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.817265 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:34.817461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.817278 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e448265-6edd-49ee-9fb1-02d58a2b7941-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:34.817461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.817289 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e448265-6edd-49ee-9fb1-02d58a2b7941-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:34.930586 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.930559 4541 generic.go:356] "Generic (PLEG): container finished" podID="0e448265-6edd-49ee-9fb1-02d58a2b7941" containerID="6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63" exitCode=0 Nov 03 14:39:34.930744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.930595 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" Nov 03 14:39:34.930744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.930617 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" event={"ID":"0e448265-6edd-49ee-9fb1-02d58a2b7941","Type":"ContainerDied","Data":"6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63"} Nov 03 14:39:34.930744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.930647 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" event={"ID":"0e448265-6edd-49ee-9fb1-02d58a2b7941","Type":"ContainerDied","Data":"fcd15a8029c5b963a285bd2c1f8b3e9561d1972ddf9bedf6f646c0c4727067a5"} Nov 03 14:39:34.930744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.930663 4541 scope.go:122] "RemoveContainer" containerID="6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63" Nov 03 14:39:34.938618 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.938592 4541 scope.go:122] "RemoveContainer" containerID="6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63" Nov 03 14:39:34.938897 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:39:34.938871 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63\": container with ID starting with 6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63 not found: ID does not exist" containerID="6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63" Nov 03 14:39:34.938981 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:34.938908 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63"} err="failed to get container status \"6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63\": rpc error: code = NotFound desc = could not find container \"6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63\": container with ID starting with 6aabf3a2b92bd9ad5d390da3c73bcad0e0a0ba995c3d8b6ee4532977f7d4bf63 not found: ID does not exist" Nov 03 14:39:35.085380 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:35.085346 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv"] Nov 03 14:39:35.089004 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:35.088977 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv"] Nov 03 14:39:35.456185 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:35.456144 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" podUID="1c42b708-0024-4dc8-855a-f2d53d6c33f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:39:35.456626 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:35.456157 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" podUID="d43d712c-24d9-4137-9a3a-fb3052623a07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:39:35.631652 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:35.631618 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-8366e3/simulator-single-main-8366e3-kserve-75c7fc7d55-jv2nv" podUID="0e448265-6edd-49ee-9fb1-02d58a2b7941" containerName="main" probeResult="failure" output="Get \"https://10.133.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 14:39:36.866300 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:36.866266 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e448265-6edd-49ee-9fb1-02d58a2b7941" path="/var/lib/kubelet/pods/0e448265-6edd-49ee-9fb1-02d58a2b7941/volumes" Nov 03 14:39:45.456591 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:45.456548 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" podUID="d43d712c-24d9-4137-9a3a-fb3052623a07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:39:45.456591 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:45.456574 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" podUID="1c42b708-0024-4dc8-855a-f2d53d6c33f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:39:46.892953 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:39:46.892920 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:39:46.928092 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.928070 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:39:46.984664 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.984637 4541 generic.go:356] "Generic (PLEG): container finished" podID="1c42b708-0024-4dc8-855a-f2d53d6c33f6" containerID="671925d7b8dcc3c67bfb47dfe3b11c623539245980c1a7479866cd47c56a358b" exitCode=0 Nov 03 14:39:46.984852 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.984710 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" event={"ID":"1c42b708-0024-4dc8-855a-f2d53d6c33f6","Type":"ContainerDied","Data":"671925d7b8dcc3c67bfb47dfe3b11c623539245980c1a7479866cd47c56a358b"} Nov 03 14:39:46.986111 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.986086 4541 generic.go:356] "Generic (PLEG): container finished" podID="d43d712c-24d9-4137-9a3a-fb3052623a07" containerID="c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9" exitCode=0 Nov 03 14:39:46.986250 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.986128 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" Nov 03 14:39:46.986250 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.986162 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" event={"ID":"d43d712c-24d9-4137-9a3a-fb3052623a07","Type":"ContainerDied","Data":"c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9"} Nov 03 14:39:46.986250 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.986210 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8" event={"ID":"d43d712c-24d9-4137-9a3a-fb3052623a07","Type":"ContainerDied","Data":"2770292d02a819a3603ef0d69dddd4b409323d93ab7d88ee0804ff745eb22a1d"} Nov 03 14:39:46.986250 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.986239 4541 scope.go:122] "RemoveContainer" containerID="c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9" Nov 03 14:39:46.994421 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.994403 4541 scope.go:122] "RemoveContainer" containerID="c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9" Nov 03 14:39:46.994693 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:39:46.994669 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9\": container with ID starting with c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9 not found: ID does not exist" containerID="c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9" Nov 03 14:39:46.994800 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:46.994700 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9"} err="failed to get container status \"c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9\": rpc error: code = NotFound desc = could not find container \"c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9\": container with ID starting with c034f8fec5555099c4837d27b077384f9b0f86e6656ec20982cfa223d62b55a9 not found: ID does not exist" Nov 03 14:39:47.023725 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.023688 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43d712c-24d9-4137-9a3a-fb3052623a07-kube-api-access-bl57q\" (UniqueName: \"kubernetes.io/projected/d43d712c-24d9-4137-9a3a-fb3052623a07-kube-api-access-bl57q\") pod \"d43d712c-24d9-4137-9a3a-fb3052623a07\" (UID: \"d43d712c-24d9-4137-9a3a-fb3052623a07\") " Nov 03 14:39:47.023876 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.023754 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43d712c-24d9-4137-9a3a-fb3052623a07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43d712c-24d9-4137-9a3a-fb3052623a07-plugins-config-volume\") pod \"d43d712c-24d9-4137-9a3a-fb3052623a07\" (UID: \"d43d712c-24d9-4137-9a3a-fb3052623a07\") " Nov 03 14:39:47.024164 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.024139 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43d712c-24d9-4137-9a3a-fb3052623a07-plugins-config-volume" pod "d43d712c-24d9-4137-9a3a-fb3052623a07" (UID: "d43d712c-24d9-4137-9a3a-fb3052623a07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:47.025840 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.025820 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43d712c-24d9-4137-9a3a-fb3052623a07-kube-api-access-bl57q" pod "d43d712c-24d9-4137-9a3a-fb3052623a07" (UID: "d43d712c-24d9-4137-9a3a-fb3052623a07"). InnerVolumeSpecName "kube-api-access-bl57q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:47.042558 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.042538 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:39:47.124456 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.124420 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c42b708-0024-4dc8-855a-f2d53d6c33f6-kube-api-access-d4j7q\" (UniqueName: \"kubernetes.io/projected/1c42b708-0024-4dc8-855a-f2d53d6c33f6-kube-api-access-d4j7q\") pod \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\" (UID: \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\") " Nov 03 14:39:47.124655 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.124523 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c42b708-0024-4dc8-855a-f2d53d6c33f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c42b708-0024-4dc8-855a-f2d53d6c33f6-plugins-config-volume\") pod \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\" (UID: \"1c42b708-0024-4dc8-855a-f2d53d6c33f6\") " Nov 03 14:39:47.124730 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.124680 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl57q\" (UniqueName: \"kubernetes.io/projected/d43d712c-24d9-4137-9a3a-fb3052623a07-kube-api-access-bl57q\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:47.124730 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.124692 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43d712c-24d9-4137-9a3a-fb3052623a07-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:47.124901 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.124877 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c42b708-0024-4dc8-855a-f2d53d6c33f6-plugins-config-volume" pod "1c42b708-0024-4dc8-855a-f2d53d6c33f6" (UID: "1c42b708-0024-4dc8-855a-f2d53d6c33f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:47.126690 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.126668 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c42b708-0024-4dc8-855a-f2d53d6c33f6-kube-api-access-d4j7q" pod "1c42b708-0024-4dc8-855a-f2d53d6c33f6" (UID: "1c42b708-0024-4dc8-855a-f2d53d6c33f6"). InnerVolumeSpecName "kube-api-access-d4j7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:47.225497 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.225427 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4j7q\" (UniqueName: \"kubernetes.io/projected/1c42b708-0024-4dc8-855a-f2d53d6c33f6-kube-api-access-d4j7q\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:47.225497 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.225454 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c42b708-0024-4dc8-855a-f2d53d6c33f6-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:47.402227 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.402194 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8"] Nov 03 14:39:47.404033 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.403999 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8366e3-65d99b9b6l5kc8"] Nov 03 14:39:47.991620 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.991541 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" Nov 03 14:39:47.992082 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.991548 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm" event={"ID":"1c42b708-0024-4dc8-855a-f2d53d6c33f6","Type":"ContainerDied","Data":"d025d0da8afe5e39d4d9086c24cbaa2cb2d327119f87d4e3e954f230ed913852"} Nov 03 14:39:47.992082 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:47.991684 4541 scope.go:122] "RemoveContainer" containerID="671925d7b8dcc3c67bfb47dfe3b11c623539245980c1a7479866cd47c56a358b" Nov 03 14:39:48.150901 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:48.150864 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm"] Nov 03 14:39:48.197663 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:48.197625 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8366e3-577465b8sm"] Nov 03 14:39:48.866903 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:48.866863 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c42b708-0024-4dc8-855a-f2d53d6c33f6" path="/var/lib/kubelet/pods/1c42b708-0024-4dc8-855a-f2d53d6c33f6/volumes" Nov 03 14:39:48.867329 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:48.867307 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43d712c-24d9-4137-9a3a-fb3052623a07" path="/var/lib/kubelet/pods/d43d712c-24d9-4137-9a3a-fb3052623a07/volumes" Nov 03 14:39:52.313673 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:52.313637 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq"] Nov 03 14:39:52.314171 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:52.313911 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" containerID="cri-o://79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980" gracePeriod=30 Nov 03 14:39:57.558020 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.557989 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:39:57.722170 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722135 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-kube-api-access-4g6vv\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-kube-api-access-4g6vv\") pod \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " Nov 03 14:39:57.722498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722191 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-certs\") pod \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " Nov 03 14:39:57.722498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722224 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-envoy\") pod \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " Nov 03 14:39:57.722498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722252 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-podinfo\") pod \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " Nov 03 14:39:57.722498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722347 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-token\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-token\") pod \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " Nov 03 14:39:57.722498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722405 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-data\") pod \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " Nov 03 14:39:57.722498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722435 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-credential-socket\") pod \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " Nov 03 14:39:57.722498 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722487 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-socket\") pod \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " Nov 03 14:39:57.722906 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722534 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istiod-ca-cert\") pod \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\" (UID: \"2616fc02-53c8-4676-a91a-0ba4e99e9c00\") " Nov 03 14:39:57.722971 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.722941 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-credential-socket" pod "2616fc02-53c8-4676-a91a-0ba4e99e9c00" (UID: "2616fc02-53c8-4676-a91a-0ba4e99e9c00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:57.723208 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.723179 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istiod-ca-cert" pod "2616fc02-53c8-4676-a91a-0ba4e99e9c00" (UID: "2616fc02-53c8-4676-a91a-0ba4e99e9c00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:57.723317 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.723231 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-data" pod "2616fc02-53c8-4676-a91a-0ba4e99e9c00" (UID: "2616fc02-53c8-4676-a91a-0ba4e99e9c00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:57.723566 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.723538 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-socket" pod "2616fc02-53c8-4676-a91a-0ba4e99e9c00" (UID: "2616fc02-53c8-4676-a91a-0ba4e99e9c00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:57.723659 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.723570 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-certs" pod "2616fc02-53c8-4676-a91a-0ba4e99e9c00" (UID: "2616fc02-53c8-4676-a91a-0ba4e99e9c00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:57.725330 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.725307 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-kube-api-access-4g6vv" pod "2616fc02-53c8-4676-a91a-0ba4e99e9c00" (UID: "2616fc02-53c8-4676-a91a-0ba4e99e9c00"). InnerVolumeSpecName "kube-api-access-4g6vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:57.725330 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.725319 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-envoy" pod "2616fc02-53c8-4676-a91a-0ba4e99e9c00" (UID: "2616fc02-53c8-4676-a91a-0ba4e99e9c00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:39:57.725454 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.725346 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-token" pod "2616fc02-53c8-4676-a91a-0ba4e99e9c00" (UID: "2616fc02-53c8-4676-a91a-0ba4e99e9c00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:57.725532 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.725514 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-podinfo" pod "2616fc02-53c8-4676-a91a-0ba4e99e9c00" (UID: "2616fc02-53c8-4676-a91a-0ba4e99e9c00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:39:57.824172 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.824138 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g6vv\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-kube-api-access-4g6vv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:57.824172 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.824169 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:57.824172 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.824178 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:57.824393 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.824187 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:57.824393 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.824196 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:57.824393 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.824204 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:57.824393 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.824212 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:57.824393 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.824220 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2616fc02-53c8-4676-a91a-0ba4e99e9c00-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:57.824393 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:57.824228 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2616fc02-53c8-4676-a91a-0ba4e99e9c00-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:58.038194 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.038115 4541 generic.go:356] "Generic (PLEG): container finished" podID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerID="79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980" exitCode=0 Nov 03 14:39:58.038194 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.038166 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" Nov 03 14:39:58.038430 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.038204 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" event={"ID":"2616fc02-53c8-4676-a91a-0ba4e99e9c00","Type":"ContainerDied","Data":"79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980"} Nov 03 14:39:58.038430 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.038269 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq" event={"ID":"2616fc02-53c8-4676-a91a-0ba4e99e9c00","Type":"ContainerDied","Data":"ff53a516c35276423279b63fa16fe8fe2870b555fb40f3c274cb1085b661faf1"} Nov 03 14:39:58.038430 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.038298 4541 scope.go:122] "RemoveContainer" containerID="79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980" Nov 03 14:39:58.047083 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.047067 4541 scope.go:122] "RemoveContainer" containerID="d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2" Nov 03 14:39:58.054425 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.054409 4541 scope.go:122] "RemoveContainer" containerID="79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980" Nov 03 14:39:58.054699 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:39:58.054669 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980\": container with ID starting with 79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980 not found: ID does not exist" containerID="79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980" Nov 03 14:39:58.054746 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.054709 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980"} err="failed to get container status \"79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980\": rpc error: code = NotFound desc = could not find container \"79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980\": container with ID starting with 79fd7293acfad136fec03f44555fc80b6f8d5fde167a335626eb924e47af1980 not found: ID does not exist" Nov 03 14:39:58.054746 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.054728 4541 scope.go:122] "RemoveContainer" containerID="d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2" Nov 03 14:39:58.054951 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:39:58.054936 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2\": container with ID starting with d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2 not found: ID does not exist" containerID="d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2" Nov 03 14:39:58.054989 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.054955 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2"} err="failed to get container status \"d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2\": rpc error: code = NotFound desc = could not find container \"d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2\": container with ID starting with d8e590558632a47cbc728a5ca43cdce5ed92fa7b2fd13860b839fe596f890fb2 not found: ID does not exist" Nov 03 14:39:58.199615 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.199579 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq"] Nov 03 14:39:58.246932 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.246903 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8366e3-openshift-default-86d478985-j6ssq"] Nov 03 14:39:58.867248 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:39:58.867217 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" path="/var/lib/kubelet/pods/2616fc02-53c8-4676-a91a-0ba4e99e9c00/volumes" Nov 03 14:40:08.306796 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.306749 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf"] Nov 03 14:40:08.307207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307129 4541 cpu_manager.go:414] "containerMap: removing container" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" Nov 03 14:40:08.307207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307142 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" Nov 03 14:40:08.307207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307163 4541 cpu_manager.go:414] "containerMap: removing container" podUID="1c42b708-0024-4dc8-855a-f2d53d6c33f6" containerName="payload-pre-processing" Nov 03 14:40:08.307207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307169 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c42b708-0024-4dc8-855a-f2d53d6c33f6" containerName="payload-pre-processing" Nov 03 14:40:08.307207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307177 4541 cpu_manager.go:414] "containerMap: removing container" podUID="01791df9-2ba7-4cde-a425-7afbbacf95e2" containerName="maas-api" Nov 03 14:40:08.307207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307182 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01791df9-2ba7-4cde-a425-7afbbacf95e2" containerName="maas-api" Nov 03 14:40:08.307207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307190 4541 cpu_manager.go:414] "containerMap: removing container" podUID="0e448265-6edd-49ee-9fb1-02d58a2b7941" containerName="main" Nov 03 14:40:08.307207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307197 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e448265-6edd-49ee-9fb1-02d58a2b7941" containerName="main" Nov 03 14:40:08.307207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307212 4541 cpu_manager.go:414] "containerMap: removing container" podUID="d43d712c-24d9-4137-9a3a-fb3052623a07" containerName="payload-processing" Nov 03 14:40:08.307524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307218 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43d712c-24d9-4137-9a3a-fb3052623a07" containerName="payload-processing" Nov 03 14:40:08.307524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307226 4541 cpu_manager.go:414] "containerMap: removing container" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" Nov 03 14:40:08.307524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307231 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" Nov 03 14:40:08.307524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307287 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c42b708-0024-4dc8-855a-f2d53d6c33f6" containerName="payload-pre-processing" Nov 03 14:40:08.307524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307297 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="01791df9-2ba7-4cde-a425-7afbbacf95e2" containerName="maas-api" Nov 03 14:40:08.307524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307302 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e448265-6edd-49ee-9fb1-02d58a2b7941" containerName="main" Nov 03 14:40:08.307524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307310 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43d712c-24d9-4137-9a3a-fb3052623a07" containerName="payload-processing" Nov 03 14:40:08.307524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307317 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" Nov 03 14:40:08.307524 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.307324 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="2616fc02-53c8-4676-a91a-0ba4e99e9c00" containerName="istio-proxy" Nov 03 14:40:08.309368 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.309348 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.406218 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.406193 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b48854d7-kserve-self-signed-certs\"" Nov 03 14:40:08.408269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.408249 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf"] Nov 03 14:40:08.414303 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.414287 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9kgj\" (UniqueName: \"kubernetes.io/projected/7a622205-5a7b-4176-907f-da8717f3667b-kube-api-access-d9kgj\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.414371 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.414339 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-tmp-dir\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.414371 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.414356 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-dshm\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.414486 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.414452 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-model-cache\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.414535 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.414520 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-home\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.414590 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.414579 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a622205-5a7b-4176-907f-da8717f3667b-tls-certs\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.516023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.515984 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9kgj\" (UniqueName: \"kubernetes.io/projected/7a622205-5a7b-4176-907f-da8717f3667b-kube-api-access-d9kgj\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.516176 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.516124 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-tmp-dir\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.516176 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.516144 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-dshm\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.516176 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.516160 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-model-cache\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.516324 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.516300 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-home\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.516379 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.516365 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a622205-5a7b-4176-907f-da8717f3667b-tls-certs\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.516635 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.516610 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-tmp-dir\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.516793 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.516647 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-model-cache\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.516793 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.516710 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-home\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.518368 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.518346 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-dshm\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.518714 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.518697 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a622205-5a7b-4176-907f-da8717f3667b-tls-certs\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.582688 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.582641 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9kgj\" (UniqueName: \"kubernetes.io/projected/7a622205-5a7b-4176-907f-da8717f3667b-kube-api-access-d9kgj\") pod \"e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.619718 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.619692 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:08.983381 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:08.983356 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf"] Nov 03 14:40:08.985359 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:40:08.985328 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a622205_5a7b_4176_907f_da8717f3667b.slice/crio-b49244943a1ef2065ee4c76ac6635bfc4b9cf2dcb7527ea42a68677bac88b85a WatchSource:0}: Error finding container b49244943a1ef2065ee4c76ac6635bfc4b9cf2dcb7527ea42a68677bac88b85a: Status 404 returned error can't find the container with id b49244943a1ef2065ee4c76ac6635bfc4b9cf2dcb7527ea42a68677bac88b85a Nov 03 14:40:09.090390 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:09.090359 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" event={"ID":"7a622205-5a7b-4176-907f-da8717f3667b","Type":"ContainerStarted","Data":"1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143"} Nov 03 14:40:09.090390 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:09.090397 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" event={"ID":"7a622205-5a7b-4176-907f-da8717f3667b","Type":"ContainerStarted","Data":"b49244943a1ef2065ee4c76ac6635bfc4b9cf2dcb7527ea42a68677bac88b85a"} Nov 03 14:40:09.205269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:09.205222 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" podStartSLOduration=1.205209277 podStartE2EDuration="1.205209277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:40:09.202694731 +0000 UTC m=+5740.516666592" watchObservedRunningTime="2026-11-03 14:40:09.205209277 +0000 UTC m=+5740.519181137" Nov 03 14:40:09.620283 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:09.620252 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:16.887926 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:40:16.887887 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:40:18.619969 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:18.619931 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:18.635935 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:18.633108 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:19.152271 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:19.152239 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:40:36.505280 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.505249 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv"] Nov 03 14:40:36.510407 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.510384 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.583343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.583310 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6jjx\" (UniqueName: \"kubernetes.io/projected/2d043adb-af5d-4392-9d77-b46bfc0610f1-kube-api-access-p6jjx\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.583343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.583353 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-tmp-dir\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.583529 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.583381 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-dshm\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.583529 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.583496 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-model-cache\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.583620 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.583529 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-home\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.583620 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.583560 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d043adb-af5d-4392-9d77-b46bfc0610f1-tls-certs\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.607324 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.607293 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b48854d7-kserve-self-signed-certs\"" Nov 03 14:40:36.612469 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.611569 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv"] Nov 03 14:40:36.684170 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.684133 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-model-cache\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.684170 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.684173 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-home\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.684400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.684296 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d043adb-af5d-4392-9d77-b46bfc0610f1-tls-certs\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.684400 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.684378 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6jjx\" (UniqueName: \"kubernetes.io/projected/2d043adb-af5d-4392-9d77-b46bfc0610f1-kube-api-access-p6jjx\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.684504 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.684404 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-tmp-dir\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.684504 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.684427 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-dshm\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.684609 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.684536 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-model-cache\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.684609 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.684574 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-home\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.684735 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.684697 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-tmp-dir\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.686567 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.686544 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-dshm\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.686981 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.686960 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d043adb-af5d-4392-9d77-b46bfc0610f1-tls-certs\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.739875 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.739853 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6jjx\" (UniqueName: \"kubernetes.io/projected/2d043adb-af5d-4392-9d77-b46bfc0610f1-kube-api-access-p6jjx\") pod \"e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:36.822070 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:36.822015 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:37.199686 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:37.199652 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv"] Nov 03 14:40:37.201344 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:40:37.201317 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d043adb_af5d_4392_9d77_b46bfc0610f1.slice/crio-69c7278d5d38f725db3fbb6bc326ef7e277dc587d65eadf43a064f5d4e8dd146 WatchSource:0}: Error finding container 69c7278d5d38f725db3fbb6bc326ef7e277dc587d65eadf43a064f5d4e8dd146: Status 404 returned error can't find the container with id 69c7278d5d38f725db3fbb6bc326ef7e277dc587d65eadf43a064f5d4e8dd146 Nov 03 14:40:37.222195 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:37.222162 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" event={"ID":"2d043adb-af5d-4392-9d77-b46bfc0610f1","Type":"ContainerStarted","Data":"69c7278d5d38f725db3fbb6bc326ef7e277dc587d65eadf43a064f5d4e8dd146"} Nov 03 14:40:38.228126 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:38.228084 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" event={"ID":"2d043adb-af5d-4392-9d77-b46bfc0610f1","Type":"ContainerStarted","Data":"f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084"} Nov 03 14:40:38.340755 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:38.340704 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" podStartSLOduration=2.340690392 podStartE2EDuration="2.340690392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:40:38.339584662 +0000 UTC m=+5769.653556533" watchObservedRunningTime="2026-11-03 14:40:38.340690392 +0000 UTC m=+5769.654662251" Nov 03 14:40:38.822951 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:38.822909 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:46.822330 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:46.822296 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:46.834602 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:46.834573 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:40:46.889035 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:40:46.889002 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:40:47.330219 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:40:47.330188 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:41:16.888400 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:41:16.888313 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:41:24.315659 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:24.315626 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf"] Nov 03 14:41:38.178387 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:38.178349 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv"] Nov 03 14:41:39.334853 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.334792 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" podUID="7a622205-5a7b-4176-907f-da8717f3667b" containerName="main" containerID="cri-o://1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143" gracePeriod=45 Nov 03 14:41:39.578389 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.578364 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:41:39.640958 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.640864 4541 generic.go:356] "Generic (PLEG): container finished" podID="7a622205-5a7b-4176-907f-da8717f3667b" containerID="1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143" exitCode=0 Nov 03 14:41:39.640958 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.640923 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" Nov 03 14:41:39.641149 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.640952 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" event={"ID":"7a622205-5a7b-4176-907f-da8717f3667b","Type":"ContainerDied","Data":"1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143"} Nov 03 14:41:39.641149 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.640991 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf" event={"ID":"7a622205-5a7b-4176-907f-da8717f3667b","Type":"ContainerDied","Data":"b49244943a1ef2065ee4c76ac6635bfc4b9cf2dcb7527ea42a68677bac88b85a"} Nov 03 14:41:39.641149 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.641008 4541 scope.go:122] "RemoveContainer" containerID="1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143" Nov 03 14:41:39.649329 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.649311 4541 scope.go:122] "RemoveContainer" containerID="1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143" Nov 03 14:41:39.649580 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:41:39.649562 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143\": container with ID starting with 1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143 not found: ID does not exist" containerID="1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143" Nov 03 14:41:39.649631 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.649590 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143"} err="failed to get container status \"1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143\": rpc error: code = NotFound desc = could not find container \"1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143\": container with ID starting with 1cdcb28778f2aa30b34754b28088d7a51e0c5a05b6e4114b44f7ee3edf685143 not found: ID does not exist" Nov 03 14:41:39.673912 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.673885 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-home\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-home\") pod \"7a622205-5a7b-4176-907f-da8717f3667b\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " Nov 03 14:41:39.674018 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.673927 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-model-cache\") pod \"7a622205-5a7b-4176-907f-da8717f3667b\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " Nov 03 14:41:39.674018 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.673957 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-dshm\") pod \"7a622205-5a7b-4176-907f-da8717f3667b\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " Nov 03 14:41:39.674126 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.674057 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-tmp-dir\") pod \"7a622205-5a7b-4176-907f-da8717f3667b\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " Nov 03 14:41:39.674126 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.674117 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-home" pod "7a622205-5a7b-4176-907f-da8717f3667b" (UID: "7a622205-5a7b-4176-907f-da8717f3667b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:41:39.674230 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.674135 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-model-cache" pod "7a622205-5a7b-4176-907f-da8717f3667b" (UID: "7a622205-5a7b-4176-907f-da8717f3667b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:41:39.674230 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.674168 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a622205-5a7b-4176-907f-da8717f3667b-kube-api-access-d9kgj\" (UniqueName: \"kubernetes.io/projected/7a622205-5a7b-4176-907f-da8717f3667b-kube-api-access-d9kgj\") pod \"7a622205-5a7b-4176-907f-da8717f3667b\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " Nov 03 14:41:39.674230 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.674220 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a622205-5a7b-4176-907f-da8717f3667b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7a622205-5a7b-4176-907f-da8717f3667b-tls-certs\") pod \"7a622205-5a7b-4176-907f-da8717f3667b\" (UID: \"7a622205-5a7b-4176-907f-da8717f3667b\") " Nov 03 14:41:39.674380 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.674280 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-tmp-dir" pod "7a622205-5a7b-4176-907f-da8717f3667b" (UID: "7a622205-5a7b-4176-907f-da8717f3667b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:41:39.674495 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.674479 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:39.674549 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.674499 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:39.674549 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.674513 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:39.676165 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.676137 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a622205-5a7b-4176-907f-da8717f3667b-tls-certs" pod "7a622205-5a7b-4176-907f-da8717f3667b" (UID: "7a622205-5a7b-4176-907f-da8717f3667b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:41:39.676165 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.676147 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-dshm" pod "7a622205-5a7b-4176-907f-da8717f3667b" (UID: "7a622205-5a7b-4176-907f-da8717f3667b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:41:39.676295 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.676197 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a622205-5a7b-4176-907f-da8717f3667b-kube-api-access-d9kgj" pod "7a622205-5a7b-4176-907f-da8717f3667b" (UID: "7a622205-5a7b-4176-907f-da8717f3667b"). InnerVolumeSpecName "kube-api-access-d9kgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:41:39.775184 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.775152 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7a622205-5a7b-4176-907f-da8717f3667b-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:39.775184 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.775180 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9kgj\" (UniqueName: \"kubernetes.io/projected/7a622205-5a7b-4176-907f-da8717f3667b-kube-api-access-d9kgj\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:39.775184 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:39.775190 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a622205-5a7b-4176-907f-da8717f3667b-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:40.056841 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:40.056808 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf"] Nov 03 14:41:40.103599 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:40.103572 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b48854d7-kserve-55bdd94489-r8zzf"] Nov 03 14:41:40.866243 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:40.866208 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a622205-5a7b-4176-907f-da8717f3667b" path="/var/lib/kubelet/pods/7a622205-5a7b-4176-907f-da8717f3667b/volumes" Nov 03 14:41:44.966930 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:44.966877 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:41:44.969017 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:44.968967 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podUID="b94a6140-9ce0-476f-9385-a2756e76c1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:41:46.889201 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:41:46.889153 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:41:53.197912 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.197852 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" podUID="2d043adb-af5d-4392-9d77-b46bfc0610f1" containerName="main" containerID="cri-o://f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084" gracePeriod=45 Nov 03 14:41:53.319749 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.319718 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" podUID="2d043adb-af5d-4392-9d77-b46bfc0610f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/ready\": dial tcp 10.133.0.158:8000: connect: connection refused" Nov 03 14:41:53.438956 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.438932 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:41:53.501086 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501031 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-dshm\") pod \"2d043adb-af5d-4392-9d77-b46bfc0610f1\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " Nov 03 14:41:53.501086 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501080 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d043adb-af5d-4392-9d77-b46bfc0610f1-kube-api-access-p6jjx\" (UniqueName: \"kubernetes.io/projected/2d043adb-af5d-4392-9d77-b46bfc0610f1-kube-api-access-p6jjx\") pod \"2d043adb-af5d-4392-9d77-b46bfc0610f1\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " Nov 03 14:41:53.501241 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501105 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-home\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-home\") pod \"2d043adb-af5d-4392-9d77-b46bfc0610f1\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " Nov 03 14:41:53.501241 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501124 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-model-cache\") pod \"2d043adb-af5d-4392-9d77-b46bfc0610f1\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " Nov 03 14:41:53.501241 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501159 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-tmp-dir\") pod \"2d043adb-af5d-4392-9d77-b46bfc0610f1\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " Nov 03 14:41:53.501241 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501192 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d043adb-af5d-4392-9d77-b46bfc0610f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d043adb-af5d-4392-9d77-b46bfc0610f1-tls-certs\") pod \"2d043adb-af5d-4392-9d77-b46bfc0610f1\" (UID: \"2d043adb-af5d-4392-9d77-b46bfc0610f1\") " Nov 03 14:41:53.501469 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501374 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-home" pod "2d043adb-af5d-4392-9d77-b46bfc0610f1" (UID: "2d043adb-af5d-4392-9d77-b46bfc0610f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:41:53.501469 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501388 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-model-cache" pod "2d043adb-af5d-4392-9d77-b46bfc0610f1" (UID: "2d043adb-af5d-4392-9d77-b46bfc0610f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:41:53.501650 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501626 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:53.501732 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501654 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:53.501732 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.501684 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-tmp-dir" pod "2d043adb-af5d-4392-9d77-b46bfc0610f1" (UID: "2d043adb-af5d-4392-9d77-b46bfc0610f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:41:53.503183 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.503158 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d043adb-af5d-4392-9d77-b46bfc0610f1-kube-api-access-p6jjx" pod "2d043adb-af5d-4392-9d77-b46bfc0610f1" (UID: "2d043adb-af5d-4392-9d77-b46bfc0610f1"). InnerVolumeSpecName "kube-api-access-p6jjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:41:53.503274 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.503187 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-dshm" pod "2d043adb-af5d-4392-9d77-b46bfc0610f1" (UID: "2d043adb-af5d-4392-9d77-b46bfc0610f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:41:53.503274 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.503230 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d043adb-af5d-4392-9d77-b46bfc0610f1-tls-certs" pod "2d043adb-af5d-4392-9d77-b46bfc0610f1" (UID: "2d043adb-af5d-4392-9d77-b46bfc0610f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:41:53.602847 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.602822 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6jjx\" (UniqueName: \"kubernetes.io/projected/2d043adb-af5d-4392-9d77-b46bfc0610f1-kube-api-access-p6jjx\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:53.602847 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.602845 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:53.602847 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.602854 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d043adb-af5d-4392-9d77-b46bfc0610f1-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:53.603056 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.602862 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d043adb-af5d-4392-9d77-b46bfc0610f1-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:53.717153 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.717123 4541 generic.go:356] "Generic (PLEG): container finished" podID="2d043adb-af5d-4392-9d77-b46bfc0610f1" containerID="f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084" exitCode=0 Nov 03 14:41:53.717325 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.717160 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" event={"ID":"2d043adb-af5d-4392-9d77-b46bfc0610f1","Type":"ContainerDied","Data":"f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084"} Nov 03 14:41:53.717325 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.717181 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" Nov 03 14:41:53.717325 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.717194 4541 scope.go:122] "RemoveContainer" containerID="f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084" Nov 03 14:41:53.717325 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.717185 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv" event={"ID":"2d043adb-af5d-4392-9d77-b46bfc0610f1","Type":"ContainerDied","Data":"69c7278d5d38f725db3fbb6bc326ef7e277dc587d65eadf43a064f5d4e8dd146"} Nov 03 14:41:53.726301 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.726240 4541 scope.go:122] "RemoveContainer" containerID="f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084" Nov 03 14:41:53.726547 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:41:53.726527 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084\": container with ID starting with f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084 not found: ID does not exist" containerID="f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084" Nov 03 14:41:53.726607 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.726555 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084"} err="failed to get container status \"f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084\": rpc error: code = NotFound desc = could not find container \"f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084\": container with ID starting with f3e3070a0fe3a522dc852fae91dad5f7743fba43bf08d08bc881ae8b6449c084 not found: ID does not exist" Nov 03 14:41:53.872523 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.872443 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv"] Nov 03 14:41:53.876795 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:53.876390 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b48854d7-kserve-bccb94885-94bmv"] Nov 03 14:41:54.867064 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:41:54.867029 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d043adb-af5d-4392-9d77-b46bfc0610f1" path="/var/lib/kubelet/pods/2d043adb-af5d-4392-9d77-b46bfc0610f1/volumes" Nov 03 14:42:16.888746 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:42:16.888709 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:42:24.777632 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.777586 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj"] Nov 03 14:42:24.779502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.779471 4541 cpu_manager.go:414] "containerMap: removing container" podUID="7a622205-5a7b-4176-907f-da8717f3667b" containerName="main" Nov 03 14:42:24.779668 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.779654 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a622205-5a7b-4176-907f-da8717f3667b" containerName="main" Nov 03 14:42:24.779790 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.779761 4541 cpu_manager.go:414] "containerMap: removing container" podUID="2d043adb-af5d-4392-9d77-b46bfc0610f1" containerName="main" Nov 03 14:42:24.779885 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.779873 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d043adb-af5d-4392-9d77-b46bfc0610f1" containerName="main" Nov 03 14:42:24.780185 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.780169 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a622205-5a7b-4176-907f-da8717f3667b" containerName="main" Nov 03 14:42:24.780287 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.780277 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d043adb-af5d-4392-9d77-b46bfc0610f1" containerName="main" Nov 03 14:42:24.784022 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.783998 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:24.834793 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.834741 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj"] Nov 03 14:42:24.921591 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.921557 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqwgv\" (UniqueName: \"kubernetes.io/projected/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-kube-api-access-bqwgv\") pod \"payload-pre-processing-6f9b5f59b6-94hzj\" (UID: \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\") " pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:24.921815 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:24.921717 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-plugins-config-volume\") pod \"payload-pre-processing-6f9b5f59b6-94hzj\" (UID: \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\") " pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:25.023106 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.023062 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-plugins-config-volume\") pod \"payload-pre-processing-6f9b5f59b6-94hzj\" (UID: \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\") " pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:25.023314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.023132 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqwgv\" (UniqueName: \"kubernetes.io/projected/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-kube-api-access-bqwgv\") pod \"payload-pre-processing-6f9b5f59b6-94hzj\" (UID: \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\") " pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:25.023836 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.023811 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-plugins-config-volume\") pod \"payload-pre-processing-6f9b5f59b6-94hzj\" (UID: \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\") " pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:25.078438 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.078371 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqwgv\" (UniqueName: \"kubernetes.io/projected/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-kube-api-access-bqwgv\") pod \"payload-pre-processing-6f9b5f59b6-94hzj\" (UID: \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\") " pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:25.097246 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.097213 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:25.459578 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.459550 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj"] Nov 03 14:42:25.461562 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:42:25.461531 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0abe27_4937_4cc5_850c_1a10fccc9c1a.slice/crio-d4ff5986a823d36b02c428a1fe07e01b70957e8337baf779be06f5e961ff6bb5 WatchSource:0}: Error finding container d4ff5986a823d36b02c428a1fe07e01b70957e8337baf779be06f5e961ff6bb5: Status 404 returned error can't find the container with id d4ff5986a823d36b02c428a1fe07e01b70957e8337baf779be06f5e961ff6bb5 Nov 03 14:42:25.875487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.875396 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" event={"ID":"8f0abe27-4937-4cc5-850c-1a10fccc9c1a","Type":"ContainerStarted","Data":"f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59"} Nov 03 14:42:25.875487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.875454 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:25.875487 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.875471 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" event={"ID":"8f0abe27-4937-4cc5-850c-1a10fccc9c1a","Type":"ContainerStarted","Data":"d4ff5986a823d36b02c428a1fe07e01b70957e8337baf779be06f5e961ff6bb5"} Nov 03 14:42:25.984522 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:25.984462 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" podStartSLOduration=1.984449166 podStartE2EDuration="1.984449166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:42:25.983246298 +0000 UTC m=+5877.297218159" watchObservedRunningTime="2026-11-03 14:42:25.984449166 +0000 UTC m=+5877.298421026" Nov 03 14:42:36.881994 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:36.881963 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:42:37.023567 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:37.023536 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h85vt"] Nov 03 14:42:39.177690 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.177656 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6bdf4df67f-gmf6f"] Nov 03 14:42:39.182455 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.182435 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:39.236427 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.236397 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6bdf4df67f-gmf6f"] Nov 03 14:42:39.244338 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.244308 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36301ff-95e7-4861-a6fb-532989f31b1f-plugins-config-volume\") pod \"payload-processing-6bdf4df67f-gmf6f\" (UID: \"a36301ff-95e7-4861-a6fb-532989f31b1f\") " pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:39.244513 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.244368 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6th\" (UniqueName: \"kubernetes.io/projected/a36301ff-95e7-4861-a6fb-532989f31b1f-kube-api-access-zb6th\") pod \"payload-processing-6bdf4df67f-gmf6f\" (UID: \"a36301ff-95e7-4861-a6fb-532989f31b1f\") " pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:39.345647 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.345616 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36301ff-95e7-4861-a6fb-532989f31b1f-plugins-config-volume\") pod \"payload-processing-6bdf4df67f-gmf6f\" (UID: \"a36301ff-95e7-4861-a6fb-532989f31b1f\") " pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:39.345853 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.345655 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6th\" (UniqueName: \"kubernetes.io/projected/a36301ff-95e7-4861-a6fb-532989f31b1f-kube-api-access-zb6th\") pod \"payload-processing-6bdf4df67f-gmf6f\" (UID: \"a36301ff-95e7-4861-a6fb-532989f31b1f\") " pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:39.346388 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.346359 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36301ff-95e7-4861-a6fb-532989f31b1f-plugins-config-volume\") pod \"payload-processing-6bdf4df67f-gmf6f\" (UID: \"a36301ff-95e7-4861-a6fb-532989f31b1f\") " pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:39.400488 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.400464 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6th\" (UniqueName: \"kubernetes.io/projected/a36301ff-95e7-4861-a6fb-532989f31b1f-kube-api-access-zb6th\") pod \"payload-processing-6bdf4df67f-gmf6f\" (UID: \"a36301ff-95e7-4861-a6fb-532989f31b1f\") " pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:39.496824 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.496742 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:39.858719 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.858689 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6bdf4df67f-gmf6f"] Nov 03 14:42:39.860670 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:42:39.860635 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36301ff_95e7_4861_a6fb_532989f31b1f.slice/crio-e3b50b56874e78ef88d2bf02be8aec84736b65314178e461a005ea5cdd82dc9b WatchSource:0}: Error finding container e3b50b56874e78ef88d2bf02be8aec84736b65314178e461a005ea5cdd82dc9b: Status 404 returned error can't find the container with id e3b50b56874e78ef88d2bf02be8aec84736b65314178e461a005ea5cdd82dc9b Nov 03 14:42:39.933813 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.933744 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" event={"ID":"a36301ff-95e7-4861-a6fb-532989f31b1f","Type":"ContainerStarted","Data":"1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892"} Nov 03 14:42:39.933913 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.933815 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" event={"ID":"a36301ff-95e7-4861-a6fb-532989f31b1f","Type":"ContainerStarted","Data":"e3b50b56874e78ef88d2bf02be8aec84736b65314178e461a005ea5cdd82dc9b"} Nov 03 14:42:39.933913 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:39.933845 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:40.043943 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:40.043878 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" podStartSLOduration=1.043862622 podStartE2EDuration="1.043862622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:42:40.043061263 +0000 UTC m=+5891.357033123" watchObservedRunningTime="2026-11-03 14:42:40.043862622 +0000 UTC m=+5891.357834481" Nov 03 14:42:42.024304 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.024256 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" podUID="392c3cf5-f8f8-4cdb-ba5c-23c0230caf27" containerName="payload-pre-processing" containerID="cri-o://623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577" gracePeriod=25 Nov 03 14:42:42.275957 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.275898 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 14:42:42.372542 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.372514 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-kube-api-access-5r42b\" (UniqueName: \"kubernetes.io/projected/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-kube-api-access-5r42b\") pod \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\" (UID: \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\") " Nov 03 14:42:42.372671 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.372654 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-plugins-config-volume\") pod \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\" (UID: \"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27\") " Nov 03 14:42:42.373011 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.372990 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-plugins-config-volume" pod "392c3cf5-f8f8-4cdb-ba5c-23c0230caf27" (UID: "392c3cf5-f8f8-4cdb-ba5c-23c0230caf27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:42:42.374739 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.374717 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-kube-api-access-5r42b" pod "392c3cf5-f8f8-4cdb-ba5c-23c0230caf27" (UID: "392c3cf5-f8f8-4cdb-ba5c-23c0230caf27"). InnerVolumeSpecName "kube-api-access-5r42b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:42:42.473479 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.473449 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:42:42.473479 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.473474 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r42b\" (UniqueName: \"kubernetes.io/projected/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27-kube-api-access-5r42b\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:42:42.947628 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.947579 4541 generic.go:356] "Generic (PLEG): container finished" podID="392c3cf5-f8f8-4cdb-ba5c-23c0230caf27" containerID="623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577" exitCode=0 Nov 03 14:42:42.947628 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.947626 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" Nov 03 14:42:42.947894 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.947689 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" event={"ID":"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27","Type":"ContainerDied","Data":"623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577"} Nov 03 14:42:42.947894 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.947729 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-h85vt" event={"ID":"392c3cf5-f8f8-4cdb-ba5c-23c0230caf27","Type":"ContainerDied","Data":"76dfbd1394c4b64caaf0df4eab771270d2c9f2ea676053d7c1dcf47ae4659dc9"} Nov 03 14:42:42.947894 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.947749 4541 scope.go:122] "RemoveContainer" containerID="623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577" Nov 03 14:42:42.956019 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.955999 4541 scope.go:122] "RemoveContainer" containerID="623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577" Nov 03 14:42:42.956249 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:42:42.956229 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577\": container with ID starting with 623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577 not found: ID does not exist" containerID="623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577" Nov 03 14:42:42.956305 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:42.956258 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577"} err="failed to get container status \"623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577\": rpc error: code = NotFound desc = could not find container \"623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577\": container with ID starting with 623a507d55434c4e22b5f8a76f9a1d99773e5a6eb08e372fafac0c51af931577 not found: ID does not exist" Nov 03 14:42:43.104041 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:43.104003 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h85vt"] Nov 03 14:42:43.140824 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:43.140746 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-h85vt"] Nov 03 14:42:44.866439 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:44.866407 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392c3cf5-f8f8-4cdb-ba5c-23c0230caf27" path="/var/lib/kubelet/pods/392c3cf5-f8f8-4cdb-ba5c-23c0230caf27/volumes" Nov 03 14:42:46.888746 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:42:46.888706 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:42:50.942169 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:50.942139 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:42:51.075618 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:51.075583 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qnx22"] Nov 03 14:42:53.299643 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.299605 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7c88cb685f-zjswf"] Nov 03 14:42:53.300228 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.300208 4541 cpu_manager.go:414] "containerMap: removing container" podUID="392c3cf5-f8f8-4cdb-ba5c-23c0230caf27" containerName="payload-pre-processing" Nov 03 14:42:53.300291 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.300232 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392c3cf5-f8f8-4cdb-ba5c-23c0230caf27" containerName="payload-pre-processing" Nov 03 14:42:53.300371 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.300357 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="392c3cf5-f8f8-4cdb-ba5c-23c0230caf27" containerName="payload-pre-processing" Nov 03 14:42:53.303401 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.303380 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:42:53.359319 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.359285 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7c88cb685f-zjswf"] Nov 03 14:42:53.368748 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.368716 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkh69\" (UniqueName: \"kubernetes.io/projected/234c192e-ace3-483d-9a9b-6d92aae5d15b-kube-api-access-mkh69\") pod \"payload-processing-7c88cb685f-zjswf\" (UID: \"234c192e-ace3-483d-9a9b-6d92aae5d15b\") " pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:42:53.368923 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.368794 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234c192e-ace3-483d-9a9b-6d92aae5d15b-plugins-config-volume\") pod \"payload-processing-7c88cb685f-zjswf\" (UID: \"234c192e-ace3-483d-9a9b-6d92aae5d15b\") " pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:42:53.469849 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.469807 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234c192e-ace3-483d-9a9b-6d92aae5d15b-plugins-config-volume\") pod \"payload-processing-7c88cb685f-zjswf\" (UID: \"234c192e-ace3-483d-9a9b-6d92aae5d15b\") " pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:42:53.470061 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.469903 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkh69\" (UniqueName: \"kubernetes.io/projected/234c192e-ace3-483d-9a9b-6d92aae5d15b-kube-api-access-mkh69\") pod \"payload-processing-7c88cb685f-zjswf\" (UID: \"234c192e-ace3-483d-9a9b-6d92aae5d15b\") " pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:42:53.470582 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.470552 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/234c192e-ace3-483d-9a9b-6d92aae5d15b-plugins-config-volume\") pod \"payload-processing-7c88cb685f-zjswf\" (UID: \"234c192e-ace3-483d-9a9b-6d92aae5d15b\") " pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:42:53.525411 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.525374 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkh69\" (UniqueName: \"kubernetes.io/projected/234c192e-ace3-483d-9a9b-6d92aae5d15b-kube-api-access-mkh69\") pod \"payload-processing-7c88cb685f-zjswf\" (UID: \"234c192e-ace3-483d-9a9b-6d92aae5d15b\") " pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:42:53.621832 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.621686 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:42:53.986008 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:53.985979 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7c88cb685f-zjswf"] Nov 03 14:42:53.988013 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:42:53.987981 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234c192e_ace3_483d_9a9b_6d92aae5d15b.slice/crio-218f9930f7e406c704cf27ae1a54a012c179e342a3ca6aa80d45fca7761302c2 WatchSource:0}: Error finding container 218f9930f7e406c704cf27ae1a54a012c179e342a3ca6aa80d45fca7761302c2: Status 404 returned error can't find the container with id 218f9930f7e406c704cf27ae1a54a012c179e342a3ca6aa80d45fca7761302c2 Nov 03 14:42:54.000517 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:54.000482 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" event={"ID":"234c192e-ace3-483d-9a9b-6d92aae5d15b","Type":"ContainerStarted","Data":"218f9930f7e406c704cf27ae1a54a012c179e342a3ca6aa80d45fca7761302c2"} Nov 03 14:42:55.006024 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:55.005988 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" event={"ID":"234c192e-ace3-483d-9a9b-6d92aae5d15b","Type":"ContainerStarted","Data":"3f9e02af303499bfb5344b6f6906a15b982d192527e20467d9a00acef9c5c3c0"} Nov 03 14:42:55.006406 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:55.006121 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:42:55.114943 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:55.114866 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" podStartSLOduration=2.114848646 podStartE2EDuration="2.114848646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:42:55.114407022 +0000 UTC m=+5906.428378882" watchObservedRunningTime="2026-11-03 14:42:55.114848646 +0000 UTC m=+5906.428820506" Nov 03 14:42:56.076325 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:56.076267 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" podUID="bf6b3624-855b-4ac2-88d4-cfa28b23a721" containerName="payload-processing" containerID="cri-o://2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044" gracePeriod=25 Nov 03 14:42:56.323600 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:56.323573 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 14:42:56.395839 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:56.395749 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6b3624-855b-4ac2-88d4-cfa28b23a721-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6b3624-855b-4ac2-88d4-cfa28b23a721-plugins-config-volume\") pod \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\" (UID: \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\") " Nov 03 14:42:56.395999 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:56.395855 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6b3624-855b-4ac2-88d4-cfa28b23a721-kube-api-access-5tgq6\" (UniqueName: \"kubernetes.io/projected/bf6b3624-855b-4ac2-88d4-cfa28b23a721-kube-api-access-5tgq6\") pod \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\" (UID: \"bf6b3624-855b-4ac2-88d4-cfa28b23a721\") " Nov 03 14:42:56.396226 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:56.396197 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6b3624-855b-4ac2-88d4-cfa28b23a721-plugins-config-volume" pod "bf6b3624-855b-4ac2-88d4-cfa28b23a721" (UID: "bf6b3624-855b-4ac2-88d4-cfa28b23a721"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:42:56.397965 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:56.397939 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6b3624-855b-4ac2-88d4-cfa28b23a721-kube-api-access-5tgq6" pod "bf6b3624-855b-4ac2-88d4-cfa28b23a721" (UID: "bf6b3624-855b-4ac2-88d4-cfa28b23a721"). InnerVolumeSpecName "kube-api-access-5tgq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:42:56.497461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:56.497423 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tgq6\" (UniqueName: \"kubernetes.io/projected/bf6b3624-855b-4ac2-88d4-cfa28b23a721-kube-api-access-5tgq6\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:42:56.497461 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:56.497455 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6b3624-855b-4ac2-88d4-cfa28b23a721-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:42:57.015916 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:57.015879 4541 generic.go:356] "Generic (PLEG): container finished" podID="bf6b3624-855b-4ac2-88d4-cfa28b23a721" containerID="2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044" exitCode=0 Nov 03 14:42:57.016091 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:57.015924 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" Nov 03 14:42:57.016091 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:57.015966 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" event={"ID":"bf6b3624-855b-4ac2-88d4-cfa28b23a721","Type":"ContainerDied","Data":"2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044"} Nov 03 14:42:57.016091 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:57.016026 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-qnx22" event={"ID":"bf6b3624-855b-4ac2-88d4-cfa28b23a721","Type":"ContainerDied","Data":"473c0a49c9291b5c395676416c7047af6879158a94ef7c5f1112b10e60d2ad99"} Nov 03 14:42:57.016091 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:57.016048 4541 scope.go:122] "RemoveContainer" containerID="2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044" Nov 03 14:42:57.024209 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:57.024188 4541 scope.go:122] "RemoveContainer" containerID="2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044" Nov 03 14:42:57.024517 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:42:57.024491 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044\": container with ID starting with 2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044 not found: ID does not exist" containerID="2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044" Nov 03 14:42:57.024573 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:57.024520 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044"} err="failed to get container status \"2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044\": rpc error: code = NotFound desc = could not find container \"2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044\": container with ID starting with 2d1a5389b9e9fc79def277708d28bfc781653f42d2ac44d61cd488155e123044 not found: ID does not exist" Nov 03 14:42:57.171847 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:57.171811 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qnx22"] Nov 03 14:42:57.219612 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:57.219564 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-qnx22"] Nov 03 14:42:58.866789 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:42:58.866734 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6b3624-855b-4ac2-88d4-cfa28b23a721" path="/var/lib/kubelet/pods/bf6b3624-855b-4ac2-88d4-cfa28b23a721/volumes" Nov 03 14:43:05.709873 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:05.709838 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj"] Nov 03 14:43:06.013552 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:06.013475 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7c88cb685f-zjswf" Nov 03 14:43:06.153694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:06.153655 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6bdf4df67f-gmf6f"] Nov 03 14:43:10.710800 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:10.710741 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" podUID="8f0abe27-4937-4cc5-850c-1a10fccc9c1a" containerName="payload-pre-processing" containerID="cri-o://f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59" gracePeriod=25 Nov 03 14:43:10.957532 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:10.957507 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:43:11.026390 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.026329 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-kube-api-access-bqwgv\" (UniqueName: \"kubernetes.io/projected/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-kube-api-access-bqwgv\") pod \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\" (UID: \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\") " Nov 03 14:43:11.026390 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.026362 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-plugins-config-volume\") pod \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\" (UID: \"8f0abe27-4937-4cc5-850c-1a10fccc9c1a\") " Nov 03 14:43:11.026759 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.026740 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-plugins-config-volume" pod "8f0abe27-4937-4cc5-850c-1a10fccc9c1a" (UID: "8f0abe27-4937-4cc5-850c-1a10fccc9c1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:11.028310 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.028281 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-kube-api-access-bqwgv" pod "8f0abe27-4937-4cc5-850c-1a10fccc9c1a" (UID: "8f0abe27-4937-4cc5-850c-1a10fccc9c1a"). InnerVolumeSpecName "kube-api-access-bqwgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:11.077527 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.077499 4541 generic.go:356] "Generic (PLEG): container finished" podID="8f0abe27-4937-4cc5-850c-1a10fccc9c1a" containerID="f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59" exitCode=0 Nov 03 14:43:11.077664 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.077541 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" Nov 03 14:43:11.077664 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.077577 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" event={"ID":"8f0abe27-4937-4cc5-850c-1a10fccc9c1a","Type":"ContainerDied","Data":"f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59"} Nov 03 14:43:11.077664 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.077639 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj" event={"ID":"8f0abe27-4937-4cc5-850c-1a10fccc9c1a","Type":"ContainerDied","Data":"d4ff5986a823d36b02c428a1fe07e01b70957e8337baf779be06f5e961ff6bb5"} Nov 03 14:43:11.077849 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.077668 4541 scope.go:122] "RemoveContainer" containerID="f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59" Nov 03 14:43:11.086419 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.086404 4541 scope.go:122] "RemoveContainer" containerID="f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59" Nov 03 14:43:11.086694 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:43:11.086674 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59\": container with ID starting with f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59 not found: ID does not exist" containerID="f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59" Nov 03 14:43:11.086757 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.086701 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59"} err="failed to get container status \"f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59\": rpc error: code = NotFound desc = could not find container \"f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59\": container with ID starting with f2d9dbb03fdf04275b71e3094ede37d9f2c9d5883810f533c394de763023ec59 not found: ID does not exist" Nov 03 14:43:11.127033 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.127006 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqwgv\" (UniqueName: \"kubernetes.io/projected/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-kube-api-access-bqwgv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:43:11.127129 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.127036 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0abe27-4937-4cc5-850c-1a10fccc9c1a-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:43:11.154118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.154092 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" podUID="a36301ff-95e7-4861-a6fb-532989f31b1f" containerName="payload-processing" containerID="cri-o://1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892" gracePeriod=25 Nov 03 14:43:11.231271 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.231245 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj"] Nov 03 14:43:11.235880 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.235847 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-6f9b5f59b6-94hzj"] Nov 03 14:43:11.385062 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.385036 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:43:11.530702 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.530666 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36301ff-95e7-4861-a6fb-532989f31b1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36301ff-95e7-4861-a6fb-532989f31b1f-plugins-config-volume\") pod \"a36301ff-95e7-4861-a6fb-532989f31b1f\" (UID: \"a36301ff-95e7-4861-a6fb-532989f31b1f\") " Nov 03 14:43:11.530702 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.530711 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36301ff-95e7-4861-a6fb-532989f31b1f-kube-api-access-zb6th\" (UniqueName: \"kubernetes.io/projected/a36301ff-95e7-4861-a6fb-532989f31b1f-kube-api-access-zb6th\") pod \"a36301ff-95e7-4861-a6fb-532989f31b1f\" (UID: \"a36301ff-95e7-4861-a6fb-532989f31b1f\") " Nov 03 14:43:11.531160 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.531134 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36301ff-95e7-4861-a6fb-532989f31b1f-plugins-config-volume" pod "a36301ff-95e7-4861-a6fb-532989f31b1f" (UID: "a36301ff-95e7-4861-a6fb-532989f31b1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:43:11.532849 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.532825 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36301ff-95e7-4861-a6fb-532989f31b1f-kube-api-access-zb6th" pod "a36301ff-95e7-4861-a6fb-532989f31b1f" (UID: "a36301ff-95e7-4861-a6fb-532989f31b1f"). InnerVolumeSpecName "kube-api-access-zb6th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:43:11.631401 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.631326 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36301ff-95e7-4861-a6fb-532989f31b1f-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:43:11.631401 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:11.631357 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb6th\" (UniqueName: \"kubernetes.io/projected/a36301ff-95e7-4861-a6fb-532989f31b1f-kube-api-access-zb6th\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:43:12.083849 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.083816 4541 generic.go:356] "Generic (PLEG): container finished" podID="a36301ff-95e7-4861-a6fb-532989f31b1f" containerID="1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892" exitCode=0 Nov 03 14:43:12.084285 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.083860 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" Nov 03 14:43:12.084285 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.083908 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" event={"ID":"a36301ff-95e7-4861-a6fb-532989f31b1f","Type":"ContainerDied","Data":"1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892"} Nov 03 14:43:12.084285 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.083966 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bdf4df67f-gmf6f" event={"ID":"a36301ff-95e7-4861-a6fb-532989f31b1f","Type":"ContainerDied","Data":"e3b50b56874e78ef88d2bf02be8aec84736b65314178e461a005ea5cdd82dc9b"} Nov 03 14:43:12.084285 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.084003 4541 scope.go:122] "RemoveContainer" containerID="1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892" Nov 03 14:43:12.093570 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.093555 4541 scope.go:122] "RemoveContainer" containerID="1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892" Nov 03 14:43:12.093805 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:43:12.093760 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892\": container with ID starting with 1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892 not found: ID does not exist" containerID="1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892" Nov 03 14:43:12.093907 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.093815 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892"} err="failed to get container status \"1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892\": rpc error: code = NotFound desc = could not find container \"1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892\": container with ID starting with 1e34abbfce7f35687ac853181ff97c12bc67b7869065274311dc15aac0c15892 not found: ID does not exist" Nov 03 14:43:12.240870 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.240837 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-6bdf4df67f-gmf6f"] Nov 03 14:43:12.288052 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.288025 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-6bdf4df67f-gmf6f"] Nov 03 14:43:12.866304 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.866272 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0abe27-4937-4cc5-850c-1a10fccc9c1a" path="/var/lib/kubelet/pods/8f0abe27-4937-4cc5-850c-1a10fccc9c1a/volumes" Nov 03 14:43:12.866634 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:12.866621 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36301ff-95e7-4861-a6fb-532989f31b1f" path="/var/lib/kubelet/pods/a36301ff-95e7-4861-a6fb-532989f31b1f/volumes" Nov 03 14:43:13.313060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313027 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp"] Nov 03 14:43:13.313451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313380 4541 cpu_manager.go:414] "containerMap: removing container" podUID="8f0abe27-4937-4cc5-850c-1a10fccc9c1a" containerName="payload-pre-processing" Nov 03 14:43:13.313451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313391 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0abe27-4937-4cc5-850c-1a10fccc9c1a" containerName="payload-pre-processing" Nov 03 14:43:13.313451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313402 4541 cpu_manager.go:414] "containerMap: removing container" podUID="bf6b3624-855b-4ac2-88d4-cfa28b23a721" containerName="payload-processing" Nov 03 14:43:13.313451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313407 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6b3624-855b-4ac2-88d4-cfa28b23a721" containerName="payload-processing" Nov 03 14:43:13.313451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313415 4541 cpu_manager.go:414] "containerMap: removing container" podUID="a36301ff-95e7-4861-a6fb-532989f31b1f" containerName="payload-processing" Nov 03 14:43:13.313451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313420 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36301ff-95e7-4861-a6fb-532989f31b1f" containerName="payload-processing" Nov 03 14:43:13.313670 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313478 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6b3624-855b-4ac2-88d4-cfa28b23a721" containerName="payload-processing" Nov 03 14:43:13.313670 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313487 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36301ff-95e7-4861-a6fb-532989f31b1f" containerName="payload-processing" Nov 03 14:43:13.313670 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.313498 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0abe27-4937-4cc5-850c-1a10fccc9c1a" containerName="payload-pre-processing" Nov 03 14:43:13.317875 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.317854 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.368149 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.368119 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 14:43:13.373395 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.373358 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp"] Nov 03 14:43:13.415212 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.415176 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 14:43:13.418210 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.418179 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 14:43:13.418210 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.418195 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rt5c8\"" Nov 03 14:43:13.433974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.433944 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 14:43:13.434108 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.434002 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 14:43:13.446522 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.446493 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc65c\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-kube-api-access-cc65c\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.446630 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.446562 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.446692 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.446674 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.446743 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.446725 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.547978 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.547944 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc65c\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-kube-api-access-cc65c\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.548128 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.548000 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.548128 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.548045 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.548128 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.548081 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.548929 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.548898 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.549069 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.548990 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.550521 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.550498 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.603196 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.603075 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc65c\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-kube-api-access-cc65c\") pod \"payload-processing-external-model-5bcd7997-mvjqp\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:13.630934 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:13.630906 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:14.006064 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:14.006041 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp"] Nov 03 14:43:14.009128 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:43:14.009081 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd98d82bb_0748_43f7_8c70_237450254e1a.slice/crio-cedbd2f5352748273f9295ef0a9e5a72d5f810654c0ed4d7d51e025b84e202c1 WatchSource:0}: Error finding container cedbd2f5352748273f9295ef0a9e5a72d5f810654c0ed4d7d51e025b84e202c1: Status 404 returned error can't find the container with id cedbd2f5352748273f9295ef0a9e5a72d5f810654c0ed4d7d51e025b84e202c1 Nov 03 14:43:14.099288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:14.099254 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" event={"ID":"d98d82bb-0748-43f7-8c70-237450254e1a","Type":"ContainerStarted","Data":"f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7"} Nov 03 14:43:14.099434 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:14.099323 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" event={"ID":"d98d82bb-0748-43f7-8c70-237450254e1a","Type":"ContainerStarted","Data":"cedbd2f5352748273f9295ef0a9e5a72d5f810654c0ed4d7d51e025b84e202c1"} Nov 03 14:43:14.099518 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:14.099448 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:14.223314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:14.223257 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" podStartSLOduration=1.223243192 podStartE2EDuration="1.223243192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:43:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:43:14.22097019 +0000 UTC m=+5925.534942051" watchObservedRunningTime="2026-11-03 14:43:14.223243192 +0000 UTC m=+5925.537215051" Nov 03 14:43:16.898872 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:43:16.898829 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:43:25.109646 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:43:25.109616 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:43:46.888749 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:43:46.888709 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:44:16.888576 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:44:16.888491 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:44:29.589197 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:44:29.589163 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:44:29.600197 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:44:29.600175 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:44:46.888394 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:44:46.888356 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:44:59.690450 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:44:59.690415 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp"] Nov 03 14:45:04.690744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:04.690705 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" podUID="d98d82bb-0748-43f7-8c70-237450254e1a" containerName="payload-processing" containerID="cri-o://f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7" gracePeriod=25 Nov 03 14:45:04.937597 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:04.937572 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:45:05.028314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.028235 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-routing-overlay-volume\") pod \"d98d82bb-0748-43f7-8c70-237450254e1a\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " Nov 03 14:45:05.028314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.028275 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-provider-credentials\") pod \"d98d82bb-0748-43f7-8c70-237450254e1a\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " Nov 03 14:45:05.028314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.028301 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-kube-api-access-cc65c\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-kube-api-access-cc65c\") pod \"d98d82bb-0748-43f7-8c70-237450254e1a\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " Nov 03 14:45:05.028615 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.028345 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-plugins-config-volume\") pod \"d98d82bb-0748-43f7-8c70-237450254e1a\" (UID: \"d98d82bb-0748-43f7-8c70-237450254e1a\") " Nov 03 14:45:05.028676 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.028654 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-routing-overlay-volume" pod "d98d82bb-0748-43f7-8c70-237450254e1a" (UID: "d98d82bb-0748-43f7-8c70-237450254e1a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:05.028832 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.028811 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-plugins-config-volume" pod "d98d82bb-0748-43f7-8c70-237450254e1a" (UID: "d98d82bb-0748-43f7-8c70-237450254e1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:05.030502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.030479 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-kube-api-access-cc65c" pod "d98d82bb-0748-43f7-8c70-237450254e1a" (UID: "d98d82bb-0748-43f7-8c70-237450254e1a"). InnerVolumeSpecName "kube-api-access-cc65c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:05.030502 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.030492 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-provider-credentials" pod "d98d82bb-0748-43f7-8c70-237450254e1a" (UID: "d98d82bb-0748-43f7-8c70-237450254e1a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:05.129596 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.129564 4541 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-provider-credentials\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:05.129596 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.129591 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc65c\" (UniqueName: \"kubernetes.io/projected/d98d82bb-0748-43f7-8c70-237450254e1a-kube-api-access-cc65c\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:05.129596 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.129601 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:05.129842 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.129610 4541 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d98d82bb-0748-43f7-8c70-237450254e1a-routing-overlay-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:05.614119 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.614083 4541 generic.go:356] "Generic (PLEG): container finished" podID="d98d82bb-0748-43f7-8c70-237450254e1a" containerID="f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7" exitCode=0 Nov 03 14:45:05.614317 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.614127 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" Nov 03 14:45:05.614317 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.614151 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" event={"ID":"d98d82bb-0748-43f7-8c70-237450254e1a","Type":"ContainerDied","Data":"f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7"} Nov 03 14:45:05.614317 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.614200 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp" event={"ID":"d98d82bb-0748-43f7-8c70-237450254e1a","Type":"ContainerDied","Data":"cedbd2f5352748273f9295ef0a9e5a72d5f810654c0ed4d7d51e025b84e202c1"} Nov 03 14:45:05.614317 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.614222 4541 scope.go:122] "RemoveContainer" containerID="f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7" Nov 03 14:45:05.623514 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.623487 4541 scope.go:122] "RemoveContainer" containerID="f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7" Nov 03 14:45:05.623823 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:45:05.623805 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7\": container with ID starting with f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7 not found: ID does not exist" containerID="f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7" Nov 03 14:45:05.623889 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.623832 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7"} err="failed to get container status \"f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7\": rpc error: code = NotFound desc = could not find container \"f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7\": container with ID starting with f31fa3d1639cc47b6c292b9c634ec1b4870f93b815d60e6dc4718a258eb791a7 not found: ID does not exist" Nov 03 14:45:05.769450 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.769415 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp"] Nov 03 14:45:05.816575 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:05.816542 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-mvjqp"] Nov 03 14:45:06.887217 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:45:06.887180 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d98d82bb-0748-43f7-8c70-237450254e1a" path="/var/lib/kubelet/pods/d98d82bb-0748-43f7-8c70-237450254e1a/volumes" Nov 03 14:45:16.892989 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:45:16.892947 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:45:46.891865 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:45:46.891819 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:46:16.888595 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:46:16.888560 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:46:46.889229 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:46:46.889192 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:47:16.888973 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:47:16.888888 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:47:43.456902 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:47:43.456859 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:47:43.457612 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:47:43.457580 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:47:46.888596 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:47:46.888560 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:48:16.888340 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:48:16.888294 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:48:46.888978 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:48:46.888938 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:49:16.888492 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:49:16.888454 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:49:29.625977 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:49:29.625948 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:49:29.628939 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:49:29.628911 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:49:46.889605 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:49:46.889570 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:50:16.887923 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:50:16.887847 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:50:23.746052 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.746012 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz"] Nov 03 14:50:23.747401 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.747360 4541 cpu_manager.go:414] "containerMap: removing container" podUID="d98d82bb-0748-43f7-8c70-237450254e1a" containerName="payload-processing" Nov 03 14:50:23.747401 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.747390 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d98d82bb-0748-43f7-8c70-237450254e1a" containerName="payload-processing" Nov 03 14:50:23.747617 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.747507 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="d98d82bb-0748-43f7-8c70-237450254e1a" containerName="payload-processing" Nov 03 14:50:23.750977 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.750952 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:23.803143 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.803106 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-21ea69-openshift-default-dockercfg-qdfhl\"" Nov 03 14:50:23.850492 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.850462 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz"] Nov 03 14:50:23.912821 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.912752 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-envoy\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:23.913044 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.912886 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcph5\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-kube-api-access-wcph5\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:23.913044 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.912957 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac55d86-03ba-416e-9f20-3dc2577d504d-istiod-ca-cert\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:23.913044 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.912987 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-podinfo\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:23.913191 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.913045 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-socket\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:23.913191 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.913065 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-certs\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:23.913191 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.913146 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-credential-socket\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:23.913387 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.913209 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-data\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:23.913387 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:23.913303 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-token\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014361 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014268 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-credential-socket\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014361 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014338 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-data\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014587 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014377 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-token\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014587 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014424 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-envoy\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014587 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014453 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcph5\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-kube-api-access-wcph5\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014583 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac55d86-03ba-416e-9f20-3dc2577d504d-istiod-ca-cert\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014646 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-podinfo\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014675 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-socket\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014704 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-certs\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.014753 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014744 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-credential-socket\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.015011 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.014827 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-data\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.015063 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.015042 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-certs\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.015063 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.015053 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-socket\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.015338 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.015314 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac55d86-03ba-416e-9f20-3dc2577d504d-istiod-ca-cert\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.016694 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.016675 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-envoy\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.017031 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.017014 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-podinfo\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.069429 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.069397 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-token\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.072423 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.072402 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcph5\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-kube-api-access-wcph5\") pod \"e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.367152 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.367073 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:24.781665 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.781636 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz"] Nov 03 14:50:24.782884 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:50:24.782858 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac55d86_03ba_416e_9f20_3dc2577d504d.slice/crio-a8cc6f03ca7699836a27946b4bd24bc2c4ef3e19181884d2b1c0b33c1f7af1fe WatchSource:0}: Error finding container a8cc6f03ca7699836a27946b4bd24bc2c4ef3e19181884d2b1c0b33c1f7af1fe: Status 404 returned error can't find the container with id a8cc6f03ca7699836a27946b4bd24bc2c4ef3e19181884d2b1c0b33c1f7af1fe Nov 03 14:50:24.785096 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.785056 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:50:24.785163 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.785122 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:50:24.785204 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:24.785173 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:50:25.271859 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:25.271825 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" event={"ID":"6ac55d86-03ba-416e-9f20-3dc2577d504d","Type":"ContainerStarted","Data":"93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe"} Nov 03 14:50:25.272057 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:25.271879 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" event={"ID":"6ac55d86-03ba-416e-9f20-3dc2577d504d","Type":"ContainerStarted","Data":"a8cc6f03ca7699836a27946b4bd24bc2c4ef3e19181884d2b1c0b33c1f7af1fe"} Nov 03 14:50:25.367423 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:25.367391 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:25.372475 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:25.372452 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:25.485209 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:25.485165 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" podStartSLOduration=2.485149974 podStartE2EDuration="2.485149974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:25.437343884 +0000 UTC m=+6356.751315744" watchObservedRunningTime="2026-11-03 14:50:25.485149974 +0000 UTC m=+6356.799121837" Nov 03 14:50:26.275916 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:26.275885 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:26.276717 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:26.276697 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:32.212864 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.212811 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g"] Nov 03 14:50:32.217540 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.217517 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:32.270630 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.270604 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g"] Nov 03 14:50:32.293525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.293499 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtkhl\" (UniqueName: \"kubernetes.io/projected/b28dc95d-50b7-4820-9944-58d6e773d164-kube-api-access-wtkhl\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g\" (UID: \"b28dc95d-50b7-4820-9944-58d6e773d164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:32.293691 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.293532 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28dc95d-50b7-4820-9944-58d6e773d164-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g\" (UID: \"b28dc95d-50b7-4820-9944-58d6e773d164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:32.315350 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.315313 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ed3a3c33-dockercfg-7w2bd\"" Nov 03 14:50:32.315350 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.315327 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-21ea69\"" Nov 03 14:50:32.394349 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.394312 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtkhl\" (UniqueName: \"kubernetes.io/projected/b28dc95d-50b7-4820-9944-58d6e773d164-kube-api-access-wtkhl\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g\" (UID: \"b28dc95d-50b7-4820-9944-58d6e773d164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:32.394349 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.394352 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28dc95d-50b7-4820-9944-58d6e773d164-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g\" (UID: \"b28dc95d-50b7-4820-9944-58d6e773d164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:32.395115 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.395096 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28dc95d-50b7-4820-9944-58d6e773d164-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g\" (UID: \"b28dc95d-50b7-4820-9944-58d6e773d164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:32.448677 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.448645 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtkhl\" (UniqueName: \"kubernetes.io/projected/b28dc95d-50b7-4820-9944-58d6e773d164-kube-api-access-wtkhl\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g\" (UID: \"b28dc95d-50b7-4820-9944-58d6e773d164\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:32.530034 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.529966 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:32.891978 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:32.891954 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g"] Nov 03 14:50:32.894187 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:50:32.894137 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28dc95d_50b7_4820_9944_58d6e773d164.slice/crio-628828fb5cc9df48d55e67e02e8bc46a9bc1552e7cd1725ad5e93aa4001dfd72 WatchSource:0}: Error finding container 628828fb5cc9df48d55e67e02e8bc46a9bc1552e7cd1725ad5e93aa4001dfd72: Status 404 returned error can't find the container with id 628828fb5cc9df48d55e67e02e8bc46a9bc1552e7cd1725ad5e93aa4001dfd72 Nov 03 14:50:33.313616 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:33.313563 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" event={"ID":"b28dc95d-50b7-4820-9944-58d6e773d164","Type":"ContainerStarted","Data":"7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a"} Nov 03 14:50:33.314054 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:33.313629 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" event={"ID":"b28dc95d-50b7-4820-9944-58d6e773d164","Type":"ContainerStarted","Data":"628828fb5cc9df48d55e67e02e8bc46a9bc1552e7cd1725ad5e93aa4001dfd72"} Nov 03 14:50:33.314054 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:33.313709 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:33.424757 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:33.424706 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" podStartSLOduration=1.424693077 podStartE2EDuration="1.424693077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:33.422951125 +0000 UTC m=+6364.736922983" watchObservedRunningTime="2026-11-03 14:50:33.424693077 +0000 UTC m=+6364.738664937" Nov 03 14:50:34.812355 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:34.812309 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9"] Nov 03 14:50:34.815676 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:34.815655 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:34.865183 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:34.865157 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-21ea69-serving-cert\"" Nov 03 14:50:34.913789 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:34.913743 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9"] Nov 03 14:50:34.917344 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:34.917311 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:34.917447 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:34.917365 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrnn\" (UniqueName: \"kubernetes.io/projected/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-kube-api-access-csrnn\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:34.917538 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:34.917505 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:34.917599 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:34.917548 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-metrics-certs\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.018479 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.018446 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.018479 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.018482 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-metrics-certs\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.018713 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.018533 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.018713 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.018555 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrnn\" (UniqueName: \"kubernetes.io/projected/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-kube-api-access-csrnn\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.018713 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:50:35.018689 4541 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-serving-cert: secret "maas-api-e2e-worker-single-main-21ea69-serving-cert" not found Nov 03 14:50:35.018904 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:50:35.018766 4541 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls podName:0858dc48-922f-401b-84a9-ab7f3c2e9ec6 nodeName:}" failed. No retries permitted until 2026-11-03 14:50:35.51874517 +0000 UTC m=+6366.832717010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls") pod "maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" (UID: "0858dc48-922f-401b-84a9-ab7f3c2e9ec6") : secret "maas-api-e2e-worker-single-main-21ea69-serving-cert" not found Nov 03 14:50:35.019127 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.019105 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.020923 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.020902 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-metrics-certs\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.079606 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.079528 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrnn\" (UniqueName: \"kubernetes.io/projected/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-kube-api-access-csrnn\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.523241 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.523213 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.525556 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.525536 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:35.727564 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:35.727536 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:36.092950 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:36.092846 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9"] Nov 03 14:50:36.095674 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:50:36.095636 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0858dc48_922f_401b_84a9_ab7f3c2e9ec6.slice/crio-a80e64fff0c31031a315ca7f4bf0699220e75f0c8613e1f07b169df69a04363d WatchSource:0}: Error finding container a80e64fff0c31031a315ca7f4bf0699220e75f0c8613e1f07b169df69a04363d: Status 404 returned error can't find the container with id a80e64fff0c31031a315ca7f4bf0699220e75f0c8613e1f07b169df69a04363d Nov 03 14:50:36.097447 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:36.097428 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:50:36.332732 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:36.332695 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" event={"ID":"0858dc48-922f-401b-84a9-ab7f3c2e9ec6","Type":"ContainerStarted","Data":"a80e64fff0c31031a315ca7f4bf0699220e75f0c8613e1f07b169df69a04363d"} Nov 03 14:50:37.338497 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:37.338461 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" event={"ID":"0858dc48-922f-401b-84a9-ab7f3c2e9ec6","Type":"ContainerStarted","Data":"589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299"} Nov 03 14:50:37.338857 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:37.338601 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:37.450918 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:37.450873 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" podStartSLOduration=2.45507725 podStartE2EDuration="3.450858962s" totalImagesPullingTime="995.781712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:37.449996089 +0000 UTC m=+6368.763967947" watchObservedRunningTime="2026-11-03 14:50:37.450858962 +0000 UTC m=+6368.764830821" Nov 03 14:50:43.361525 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:43.361491 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:50:44.321094 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:44.321057 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:50:46.887725 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:50:46.887691 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:50:50.403125 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:50.403091 4541 generic.go:356] "Generic (PLEG): container finished" podID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerID="93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe" exitCode=0 Nov 03 14:50:50.403125 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:50.403133 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" event={"ID":"6ac55d86-03ba-416e-9f20-3dc2577d504d","Type":"ContainerDied","Data":"93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe"} Nov 03 14:50:50.403726 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:50.403526 4541 scope.go:122] "RemoveContainer" containerID="93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe" Nov 03 14:50:50.404676 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:50.404646 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:50:50.404740 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:50.404714 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:50:50.404795 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:50.404744 4541 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Nov 03 14:50:51.409191 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:51.409157 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" event={"ID":"6ac55d86-03ba-416e-9f20-3dc2577d504d","Type":"ContainerStarted","Data":"b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a"} Nov 03 14:50:52.368162 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:52.368128 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:52.369286 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:52.369263 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:50:52.369372 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:52.369317 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 14:50:53.367645 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:53.367607 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:50:53.368096 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:53.367663 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 14:50:54.367644 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:54.367610 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:50:54.367835 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:54.367663 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 14:50:55.367744 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:55.367707 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:50:55.367957 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:55.367760 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 14:50:56.276238 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:56.276204 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:56.367435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:56.367402 4541 patch_prober.go:29] interesting pod/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 03 14:50:56.367603 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:56.367467 4541 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 03 14:50:57.371314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:57.371282 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:57.372361 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:57.372334 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:50:58.272520 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.272484 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd"] Nov 03 14:50:58.275956 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.275934 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.325929 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.325899 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-21ea69\"/\"openshift-service-ca.crt\"" Nov 03 14:50:58.325929 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.325909 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-21ea69\"/\"kube-root-ca.crt\"" Nov 03 14:50:58.345111 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.345079 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-model-cache\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.345258 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.345152 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-dshm\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.345258 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.345179 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46203ca-461e-4838-a6a1-89e8536bf290-tls-certs\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.345258 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.345208 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-home\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.345436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.345381 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-tmp-dir\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.345529 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.345512 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8vzd\" (UniqueName: \"kubernetes.io/projected/b46203ca-461e-4838-a6a1-89e8536bf290-kube-api-access-p8vzd\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.358165 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.358145 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-21ea69\"/\"default-dockercfg-pjtgx\"" Nov 03 14:50:58.372153 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.372135 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-21ea69\"/\"simulator-single-main-21ea69-kserve-self-signed-certs\"" Nov 03 14:50:58.374140 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.374119 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd"] Nov 03 14:50:58.446435 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.446400 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-tmp-dir\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.446617 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.446500 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8vzd\" (UniqueName: \"kubernetes.io/projected/b46203ca-461e-4838-a6a1-89e8536bf290-kube-api-access-p8vzd\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.446617 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.446533 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-model-cache\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.446617 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.446578 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-dshm\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.446617 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.446602 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46203ca-461e-4838-a6a1-89e8536bf290-tls-certs\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.446862 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.446628 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-home\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.446939 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.446916 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-tmp-dir\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.447023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.447004 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-model-cache\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.447215 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.447195 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-home\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.448902 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.448883 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-dshm\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.449164 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.449144 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46203ca-461e-4838-a6a1-89e8536bf290-tls-certs\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.502087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.502063 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8vzd\" (UniqueName: \"kubernetes.io/projected/b46203ca-461e-4838-a6a1-89e8536bf290-kube-api-access-p8vzd\") pod \"simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.587942 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.587884 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:58.957273 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:58.957249 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd"] Nov 03 14:50:58.959319 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:50:58.959290 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb46203ca_461e_4838_a6a1_89e8536bf290.slice/crio-f673d86fbe2fc01a551fa4d10f5ed5c82e358e1f5c79f9136b9208d40f41bb0f WatchSource:0}: Error finding container f673d86fbe2fc01a551fa4d10f5ed5c82e358e1f5c79f9136b9208d40f41bb0f: Status 404 returned error can't find the container with id f673d86fbe2fc01a551fa4d10f5ed5c82e358e1f5c79f9136b9208d40f41bb0f Nov 03 14:50:59.447989 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:59.447953 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" event={"ID":"b46203ca-461e-4838-a6a1-89e8536bf290","Type":"ContainerStarted","Data":"210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757"} Nov 03 14:50:59.448361 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:59.447995 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" event={"ID":"b46203ca-461e-4838-a6a1-89e8536bf290","Type":"ContainerStarted","Data":"f673d86fbe2fc01a551fa4d10f5ed5c82e358e1f5c79f9136b9208d40f41bb0f"} Nov 03 14:50:59.588087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:59.588058 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:50:59.588568 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:50:59.588527 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" podStartSLOduration=1.588515336 podStartE2EDuration="1.588515336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:59.586920088 +0000 UTC m=+6390.900891946" watchObservedRunningTime="2026-11-03 14:50:59.588515336 +0000 UTC m=+6390.902487196" Nov 03 14:51:08.589083 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:51:08.589048 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:51:08.600941 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:51:08.600916 4541 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:51:09.501706 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:51:09.501673 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:51:16.888737 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:51:16.888698 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:51:46.888764 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:51:46.888715 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:52:04.739235 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:04.739199 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd"] Nov 03 14:52:06.037118 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:06.037085 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g"] Nov 03 14:52:06.443422 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:06.443390 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9"] Nov 03 14:52:06.443614 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:06.443579 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" podUID="0858dc48-922f-401b-84a9-ab7f3c2e9ec6" containerName="maas-api" containerID="cri-o://589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299" gracePeriod=30 Nov 03 14:52:07.688852 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.688828 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:52:07.748380 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.748350 4541 generic.go:356] "Generic (PLEG): container finished" podID="0858dc48-922f-401b-84a9-ab7f3c2e9ec6" containerID="589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299" exitCode=0 Nov 03 14:52:07.748519 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.748390 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" Nov 03 14:52:07.748519 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.748438 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" event={"ID":"0858dc48-922f-401b-84a9-ab7f3c2e9ec6","Type":"ContainerDied","Data":"589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299"} Nov 03 14:52:07.748519 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.748486 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9" event={"ID":"0858dc48-922f-401b-84a9-ab7f3c2e9ec6","Type":"ContainerDied","Data":"a80e64fff0c31031a315ca7f4bf0699220e75f0c8613e1f07b169df69a04363d"} Nov 03 14:52:07.748519 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.748498 4541 scope.go:122] "RemoveContainer" containerID="589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299" Nov 03 14:52:07.756653 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.756636 4541 scope.go:122] "RemoveContainer" containerID="589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299" Nov 03 14:52:07.756941 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:52:07.756922 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299\": container with ID starting with 589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299 not found: ID does not exist" containerID="589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299" Nov 03 14:52:07.756996 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.756950 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299"} err="failed to get container status \"589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299\": rpc error: code = NotFound desc = could not find container \"589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299\": container with ID starting with 589fa999c82e0a83672db18bae2658d8e7c4db90416482fb69a8c2a141093299 not found: ID does not exist" Nov 03 14:52:07.792378 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.792321 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls\") pod \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " Nov 03 14:52:07.792378 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.792373 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-kube-api-access-csrnn\" (UniqueName: \"kubernetes.io/projected/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-kube-api-access-csrnn\") pod \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " Nov 03 14:52:07.792539 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.792393 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-metrics-certs\") pod \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " Nov 03 14:52:07.792539 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.792450 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-openshift-service-ca\") pod \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\" (UID: \"0858dc48-922f-401b-84a9-ab7f3c2e9ec6\") " Nov 03 14:52:07.792865 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.792838 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-openshift-service-ca" pod "0858dc48-922f-401b-84a9-ab7f3c2e9ec6" (UID: "0858dc48-922f-401b-84a9-ab7f3c2e9ec6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:52:07.794552 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.794526 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-metrics-certs" pod "0858dc48-922f-401b-84a9-ab7f3c2e9ec6" (UID: "0858dc48-922f-401b-84a9-ab7f3c2e9ec6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:52:07.794646 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.794623 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls" pod "0858dc48-922f-401b-84a9-ab7f3c2e9ec6" (UID: "0858dc48-922f-401b-84a9-ab7f3c2e9ec6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:52:07.794696 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.794654 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-kube-api-access-csrnn" pod "0858dc48-922f-401b-84a9-ab7f3c2e9ec6" (UID: "0858dc48-922f-401b-84a9-ab7f3c2e9ec6"). InnerVolumeSpecName "kube-api-access-csrnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:52:07.892987 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.892955 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:07.892987 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.892982 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:07.892987 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.892992 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csrnn\" (UniqueName: \"kubernetes.io/projected/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-kube-api-access-csrnn\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:07.893176 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:07.893001 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0858dc48-922f-401b-84a9-ab7f3c2e9ec6-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:08.164039 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:08.164004 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9"] Nov 03 14:52:08.170038 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:08.170011 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-w8hf9"] Nov 03 14:52:08.867191 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:08.867152 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0858dc48-922f-401b-84a9-ab7f3c2e9ec6" path="/var/lib/kubelet/pods/0858dc48-922f-401b-84a9-ab7f3c2e9ec6/volumes" Nov 03 14:52:11.038156 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:11.038117 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" podUID="b28dc95d-50b7-4820-9944-58d6e773d164" containerName="payload-pre-processing" containerID="cri-o://7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a" gracePeriod=25 Nov 03 14:52:14.320258 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:14.320215 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" podUID="b28dc95d-50b7-4820-9944-58d6e773d164" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:52:16.890511 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:52:16.890469 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:52:19.758962 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:19.758922 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" podUID="b46203ca-461e-4838-a6a1-89e8536bf290" containerName="main" containerID="cri-o://210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757" gracePeriod=45 Nov 03 14:52:20.012155 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.012100 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:52:20.102572 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.102535 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-tmp-dir\") pod \"b46203ca-461e-4838-a6a1-89e8536bf290\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " Nov 03 14:52:20.102800 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.102586 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-dshm\") pod \"b46203ca-461e-4838-a6a1-89e8536bf290\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " Nov 03 14:52:20.102800 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.102638 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46203ca-461e-4838-a6a1-89e8536bf290-tls-certs\" (UniqueName: \"kubernetes.io/secret/b46203ca-461e-4838-a6a1-89e8536bf290-tls-certs\") pod \"b46203ca-461e-4838-a6a1-89e8536bf290\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " Nov 03 14:52:20.102800 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.102700 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46203ca-461e-4838-a6a1-89e8536bf290-kube-api-access-p8vzd\" (UniqueName: \"kubernetes.io/projected/b46203ca-461e-4838-a6a1-89e8536bf290-kube-api-access-p8vzd\") pod \"b46203ca-461e-4838-a6a1-89e8536bf290\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " Nov 03 14:52:20.102800 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.102767 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-home\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-home\") pod \"b46203ca-461e-4838-a6a1-89e8536bf290\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " Nov 03 14:52:20.103063 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.102861 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-model-cache\") pod \"b46203ca-461e-4838-a6a1-89e8536bf290\" (UID: \"b46203ca-461e-4838-a6a1-89e8536bf290\") " Nov 03 14:52:20.103063 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.102899 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-tmp-dir" pod "b46203ca-461e-4838-a6a1-89e8536bf290" (UID: "b46203ca-461e-4838-a6a1-89e8536bf290"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:52:20.103169 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.103082 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-home" pod "b46203ca-461e-4838-a6a1-89e8536bf290" (UID: "b46203ca-461e-4838-a6a1-89e8536bf290"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:52:20.103169 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.103143 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-model-cache" pod "b46203ca-461e-4838-a6a1-89e8536bf290" (UID: "b46203ca-461e-4838-a6a1-89e8536bf290"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:52:20.103337 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.103266 4541 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-tmp-dir\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:20.103337 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.103281 4541 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-home\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:20.103337 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.103290 4541 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-model-cache\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:20.105106 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.105083 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-dshm" pod "b46203ca-461e-4838-a6a1-89e8536bf290" (UID: "b46203ca-461e-4838-a6a1-89e8536bf290"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:52:20.105210 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.105125 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46203ca-461e-4838-a6a1-89e8536bf290-tls-certs" pod "b46203ca-461e-4838-a6a1-89e8536bf290" (UID: "b46203ca-461e-4838-a6a1-89e8536bf290"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:52:20.105210 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.105140 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46203ca-461e-4838-a6a1-89e8536bf290-kube-api-access-p8vzd" pod "b46203ca-461e-4838-a6a1-89e8536bf290" (UID: "b46203ca-461e-4838-a6a1-89e8536bf290"). InnerVolumeSpecName "kube-api-access-p8vzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:52:20.204204 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.204170 4541 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b46203ca-461e-4838-a6a1-89e8536bf290-dshm\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:20.204204 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.204198 4541 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b46203ca-461e-4838-a6a1-89e8536bf290-tls-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:20.204204 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.204208 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8vzd\" (UniqueName: \"kubernetes.io/projected/b46203ca-461e-4838-a6a1-89e8536bf290-kube-api-access-p8vzd\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:20.815531 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.815494 4541 generic.go:356] "Generic (PLEG): container finished" podID="b46203ca-461e-4838-a6a1-89e8536bf290" containerID="210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757" exitCode=0 Nov 03 14:52:20.815981 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.815535 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" Nov 03 14:52:20.815981 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.815577 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" event={"ID":"b46203ca-461e-4838-a6a1-89e8536bf290","Type":"ContainerDied","Data":"210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757"} Nov 03 14:52:20.815981 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.815619 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd" event={"ID":"b46203ca-461e-4838-a6a1-89e8536bf290","Type":"ContainerDied","Data":"f673d86fbe2fc01a551fa4d10f5ed5c82e358e1f5c79f9136b9208d40f41bb0f"} Nov 03 14:52:20.815981 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.815636 4541 scope.go:122] "RemoveContainer" containerID="210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757" Nov 03 14:52:20.825087 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.825070 4541 scope.go:122] "RemoveContainer" containerID="210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757" Nov 03 14:52:20.825327 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:52:20.825305 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757\": container with ID starting with 210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757 not found: ID does not exist" containerID="210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757" Nov 03 14:52:20.825385 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.825336 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757"} err="failed to get container status \"210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757\": rpc error: code = NotFound desc = could not find container \"210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757\": container with ID starting with 210ea384382f3dfeaa36e2e6fd4408b8dcea3da7f34420b2bb0bd2084cda8757 not found: ID does not exist" Nov 03 14:52:20.970880 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:20.970851 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd"] Nov 03 14:52:21.018010 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:21.017978 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-21ea69/simulator-single-main-21ea69-kserve-5b95dd5b44-c2wjd"] Nov 03 14:52:22.866565 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:22.866522 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46203ca-461e-4838-a6a1-89e8536bf290" path="/var/lib/kubelet/pods/b46203ca-461e-4838-a6a1-89e8536bf290/volumes" Nov 03 14:52:24.319178 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:24.319138 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" podUID="b28dc95d-50b7-4820-9944-58d6e773d164" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:52:31.286407 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.286380 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:52:31.407519 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.407483 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b28dc95d-50b7-4820-9944-58d6e773d164-kube-api-access-wtkhl\" (UniqueName: \"kubernetes.io/projected/b28dc95d-50b7-4820-9944-58d6e773d164-kube-api-access-wtkhl\") pod \"b28dc95d-50b7-4820-9944-58d6e773d164\" (UID: \"b28dc95d-50b7-4820-9944-58d6e773d164\") " Nov 03 14:52:31.407673 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.407579 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b28dc95d-50b7-4820-9944-58d6e773d164-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28dc95d-50b7-4820-9944-58d6e773d164-plugins-config-volume\") pod \"b28dc95d-50b7-4820-9944-58d6e773d164\" (UID: \"b28dc95d-50b7-4820-9944-58d6e773d164\") " Nov 03 14:52:31.407991 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.407966 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b28dc95d-50b7-4820-9944-58d6e773d164-plugins-config-volume" pod "b28dc95d-50b7-4820-9944-58d6e773d164" (UID: "b28dc95d-50b7-4820-9944-58d6e773d164"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:52:31.409618 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.409592 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28dc95d-50b7-4820-9944-58d6e773d164-kube-api-access-wtkhl" pod "b28dc95d-50b7-4820-9944-58d6e773d164" (UID: "b28dc95d-50b7-4820-9944-58d6e773d164"). InnerVolumeSpecName "kube-api-access-wtkhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:52:31.509220 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.509181 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtkhl\" (UniqueName: \"kubernetes.io/projected/b28dc95d-50b7-4820-9944-58d6e773d164-kube-api-access-wtkhl\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:31.509220 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.509215 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b28dc95d-50b7-4820-9944-58d6e773d164-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:31.866207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.866116 4541 generic.go:356] "Generic (PLEG): container finished" podID="b28dc95d-50b7-4820-9944-58d6e773d164" containerID="7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a" exitCode=0 Nov 03 14:52:31.866207 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.866162 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" Nov 03 14:52:31.866451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.866206 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" event={"ID":"b28dc95d-50b7-4820-9944-58d6e773d164","Type":"ContainerDied","Data":"7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a"} Nov 03 14:52:31.866451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.866266 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g" event={"ID":"b28dc95d-50b7-4820-9944-58d6e773d164","Type":"ContainerDied","Data":"628828fb5cc9df48d55e67e02e8bc46a9bc1552e7cd1725ad5e93aa4001dfd72"} Nov 03 14:52:31.866451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.866293 4541 scope.go:122] "RemoveContainer" containerID="7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a" Nov 03 14:52:31.875427 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.875407 4541 scope.go:122] "RemoveContainer" containerID="7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a" Nov 03 14:52:31.875686 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:52:31.875666 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a\": container with ID starting with 7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a not found: ID does not exist" containerID="7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a" Nov 03 14:52:31.875737 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:31.875696 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a"} err="failed to get container status \"7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a\": rpc error: code = NotFound desc = could not find container \"7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a\": container with ID starting with 7833df079f2d427a1e326d72f983a124e375a6cef90296ab8bd24b09f3941c3a not found: ID does not exist" Nov 03 14:52:32.021727 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:32.021685 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g"] Nov 03 14:52:32.025820 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:32.025791 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfc7r8g"] Nov 03 14:52:32.866820 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:32.866783 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b28dc95d-50b7-4820-9944-58d6e773d164" path="/var/lib/kubelet/pods/b28dc95d-50b7-4820-9944-58d6e773d164/volumes" Nov 03 14:52:44.469597 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.469552 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl"] Nov 03 14:52:44.470253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.470016 4541 cpu_manager.go:414] "containerMap: removing container" podUID="0858dc48-922f-401b-84a9-ab7f3c2e9ec6" containerName="maas-api" Nov 03 14:52:44.470253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.470032 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0858dc48-922f-401b-84a9-ab7f3c2e9ec6" containerName="maas-api" Nov 03 14:52:44.470253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.470051 4541 cpu_manager.go:414] "containerMap: removing container" podUID="b28dc95d-50b7-4820-9944-58d6e773d164" containerName="payload-pre-processing" Nov 03 14:52:44.470253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.470057 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b28dc95d-50b7-4820-9944-58d6e773d164" containerName="payload-pre-processing" Nov 03 14:52:44.470253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.470075 4541 cpu_manager.go:414] "containerMap: removing container" podUID="b46203ca-461e-4838-a6a1-89e8536bf290" containerName="main" Nov 03 14:52:44.470253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.470080 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46203ca-461e-4838-a6a1-89e8536bf290" containerName="main" Nov 03 14:52:44.470253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.470145 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="0858dc48-922f-401b-84a9-ab7f3c2e9ec6" containerName="maas-api" Nov 03 14:52:44.470253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.470155 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="b28dc95d-50b7-4820-9944-58d6e773d164" containerName="payload-pre-processing" Nov 03 14:52:44.470253 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.470165 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46203ca-461e-4838-a6a1-89e8536bf290" containerName="main" Nov 03 14:52:44.475436 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.475415 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:44.526990 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.526954 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl"] Nov 03 14:52:44.571717 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.571689 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-21ea69\"" Nov 03 14:52:44.571978 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.571942 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ed3a3c33-dockercfg-b6w89\"" Nov 03 14:52:44.628763 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.628733 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ba2e92-ab11-4edc-84a0-1339723e2f69-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl\" (UID: \"17ba2e92-ab11-4edc-84a0-1339723e2f69\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:44.628902 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.628785 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbfb\" (UniqueName: \"kubernetes.io/projected/17ba2e92-ab11-4edc-84a0-1339723e2f69-kube-api-access-mlbfb\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl\" (UID: \"17ba2e92-ab11-4edc-84a0-1339723e2f69\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:44.649324 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.649300 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz"] Nov 03 14:52:44.652895 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.652876 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:52:44.706107 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.706085 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz"] Nov 03 14:52:44.729499 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.729437 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ba2e92-ab11-4edc-84a0-1339723e2f69-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl\" (UID: \"17ba2e92-ab11-4edc-84a0-1339723e2f69\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:44.729499 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.729470 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbfb\" (UniqueName: \"kubernetes.io/projected/17ba2e92-ab11-4edc-84a0-1339723e2f69-kube-api-access-mlbfb\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl\" (UID: \"17ba2e92-ab11-4edc-84a0-1339723e2f69\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:44.730404 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.730379 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ba2e92-ab11-4edc-84a0-1339723e2f69-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl\" (UID: \"17ba2e92-ab11-4edc-84a0-1339723e2f69\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:44.783935 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.783909 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbfb\" (UniqueName: \"kubernetes.io/projected/17ba2e92-ab11-4edc-84a0-1339723e2f69-kube-api-access-mlbfb\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl\" (UID: \"17ba2e92-ab11-4edc-84a0-1339723e2f69\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:44.790735 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.790716 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:44.830163 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.830132 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz\" (UID: \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:52:44.830289 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.830171 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288tv\" (UniqueName: \"kubernetes.io/projected/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-kube-api-access-288tv\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz\" (UID: \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:52:44.930899 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.930867 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz\" (UID: \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:52:44.931093 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.930912 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288tv\" (UniqueName: \"kubernetes.io/projected/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-kube-api-access-288tv\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz\" (UID: \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:52:44.931594 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.931566 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz\" (UID: \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:52:44.988381 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:44.988268 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-288tv\" (UniqueName: \"kubernetes.io/projected/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-kube-api-access-288tv\") pod \"payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz\" (UID: \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:52:45.152373 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:45.152345 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl"] Nov 03 14:52:45.154418 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:52:45.154383 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ba2e92_ab11_4edc_84a0_1339723e2f69.slice/crio-5a417446d9ba45bb7f9306d829b83a75add6e5cc575e14b34042dabcf96289c6 WatchSource:0}: Error finding container 5a417446d9ba45bb7f9306d829b83a75add6e5cc575e14b34042dabcf96289c6: Status 404 returned error can't find the container with id 5a417446d9ba45bb7f9306d829b83a75add6e5cc575e14b34042dabcf96289c6 Nov 03 14:52:45.265559 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:45.265500 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:52:45.634366 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:45.634322 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz"] Nov 03 14:52:45.634927 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:52:45.634901 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dccd27b_49c7_4bf1_8973_f28a0dc13c35.slice/crio-7474f76ea20e9de339b94c8840eb4ece9b7a2068cd696bda2b3c307210657802 WatchSource:0}: Error finding container 7474f76ea20e9de339b94c8840eb4ece9b7a2068cd696bda2b3c307210657802: Status 404 returned error can't find the container with id 7474f76ea20e9de339b94c8840eb4ece9b7a2068cd696bda2b3c307210657802 Nov 03 14:52:45.925382 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:45.925339 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" event={"ID":"17ba2e92-ab11-4edc-84a0-1339723e2f69","Type":"ContainerStarted","Data":"30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded"} Nov 03 14:52:45.925602 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:45.925403 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" event={"ID":"17ba2e92-ab11-4edc-84a0-1339723e2f69","Type":"ContainerStarted","Data":"5a417446d9ba45bb7f9306d829b83a75add6e5cc575e14b34042dabcf96289c6"} Nov 03 14:52:45.925602 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:45.925433 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:45.926911 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:45.926884 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" event={"ID":"7dccd27b-49c7-4bf1-8973-f28a0dc13c35","Type":"ContainerStarted","Data":"0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698"} Nov 03 14:52:45.927003 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:45.926927 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" event={"ID":"7dccd27b-49c7-4bf1-8973-f28a0dc13c35","Type":"ContainerStarted","Data":"7474f76ea20e9de339b94c8840eb4ece9b7a2068cd696bda2b3c307210657802"} Nov 03 14:52:45.927053 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:45.927010 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:52:46.034616 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:46.034344 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" podStartSLOduration=2.034330361 podStartE2EDuration="2.034330361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:46.032681487 +0000 UTC m=+6497.346653347" watchObservedRunningTime="2026-11-03 14:52:46.034330361 +0000 UTC m=+6497.348302242" Nov 03 14:52:46.141675 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:46.141621 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" podStartSLOduration=2.141606594 podStartE2EDuration="2.141606594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:46.139370693 +0000 UTC m=+6497.453342552" watchObservedRunningTime="2026-11-03 14:52:46.141606594 +0000 UTC m=+6497.455578451" Nov 03 14:52:46.889300 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:52:46.889262 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:52:56.934388 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:56.934350 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:52:56.934902 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:52:56.934428 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:53:09.183875 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:09.183837 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl"] Nov 03 14:53:09.298338 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:09.298307 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz"] Nov 03 14:53:14.184248 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.184160 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" podUID="17ba2e92-ab11-4edc-84a0-1339723e2f69" containerName="payload-processing" containerID="cri-o://30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded" gracePeriod=25 Nov 03 14:53:14.299287 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.299228 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" podUID="7dccd27b-49c7-4bf1-8973-f28a0dc13c35" containerName="payload-pre-processing" containerID="cri-o://0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698" gracePeriod=25 Nov 03 14:53:14.428288 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.428264 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:53:14.496726 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.496699 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ba2e92-ab11-4edc-84a0-1339723e2f69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ba2e92-ab11-4edc-84a0-1339723e2f69-plugins-config-volume\") pod \"17ba2e92-ab11-4edc-84a0-1339723e2f69\" (UID: \"17ba2e92-ab11-4edc-84a0-1339723e2f69\") " Nov 03 14:53:14.496932 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.496745 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ba2e92-ab11-4edc-84a0-1339723e2f69-kube-api-access-mlbfb\" (UniqueName: \"kubernetes.io/projected/17ba2e92-ab11-4edc-84a0-1339723e2f69-kube-api-access-mlbfb\") pod \"17ba2e92-ab11-4edc-84a0-1339723e2f69\" (UID: \"17ba2e92-ab11-4edc-84a0-1339723e2f69\") " Nov 03 14:53:14.497175 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.497146 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ba2e92-ab11-4edc-84a0-1339723e2f69-plugins-config-volume" pod "17ba2e92-ab11-4edc-84a0-1339723e2f69" (UID: "17ba2e92-ab11-4edc-84a0-1339723e2f69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:53:14.499083 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.499053 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ba2e92-ab11-4edc-84a0-1339723e2f69-kube-api-access-mlbfb" pod "17ba2e92-ab11-4edc-84a0-1339723e2f69" (UID: "17ba2e92-ab11-4edc-84a0-1339723e2f69"). InnerVolumeSpecName "kube-api-access-mlbfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:53:14.532970 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.532952 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:53:14.598278 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.598253 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-plugins-config-volume\") pod \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\" (UID: \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\") " Nov 03 14:53:14.598416 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.598351 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-kube-api-access-288tv\" (UniqueName: \"kubernetes.io/projected/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-kube-api-access-288tv\") pod \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\" (UID: \"7dccd27b-49c7-4bf1-8973-f28a0dc13c35\") " Nov 03 14:53:14.598656 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.598636 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17ba2e92-ab11-4edc-84a0-1339723e2f69-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:14.598699 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.598663 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbfb\" (UniqueName: \"kubernetes.io/projected/17ba2e92-ab11-4edc-84a0-1339723e2f69-kube-api-access-mlbfb\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:14.598699 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.598664 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-plugins-config-volume" pod "7dccd27b-49c7-4bf1-8973-f28a0dc13c35" (UID: "7dccd27b-49c7-4bf1-8973-f28a0dc13c35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:53:14.600311 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.600289 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-kube-api-access-288tv" pod "7dccd27b-49c7-4bf1-8973-f28a0dc13c35" (UID: "7dccd27b-49c7-4bf1-8973-f28a0dc13c35"). InnerVolumeSpecName "kube-api-access-288tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:53:14.699892 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.699863 4541 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-plugins-config-volume\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:14.699892 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:14.699887 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-288tv\" (UniqueName: \"kubernetes.io/projected/7dccd27b-49c7-4bf1-8973-f28a0dc13c35-kube-api-access-288tv\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:15.057619 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.057582 4541 generic.go:356] "Generic (PLEG): container finished" podID="7dccd27b-49c7-4bf1-8973-f28a0dc13c35" containerID="0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698" exitCode=0 Nov 03 14:53:15.057817 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.057640 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" Nov 03 14:53:15.057817 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.057673 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" event={"ID":"7dccd27b-49c7-4bf1-8973-f28a0dc13c35","Type":"ContainerDied","Data":"0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698"} Nov 03 14:53:15.057817 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.057729 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz" event={"ID":"7dccd27b-49c7-4bf1-8973-f28a0dc13c35","Type":"ContainerDied","Data":"7474f76ea20e9de339b94c8840eb4ece9b7a2068cd696bda2b3c307210657802"} Nov 03 14:53:15.057817 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.057762 4541 scope.go:122] "RemoveContainer" containerID="0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698" Nov 03 14:53:15.059011 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.058991 4541 generic.go:356] "Generic (PLEG): container finished" podID="17ba2e92-ab11-4edc-84a0-1339723e2f69" containerID="30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded" exitCode=0 Nov 03 14:53:15.059130 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.059024 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" event={"ID":"17ba2e92-ab11-4edc-84a0-1339723e2f69","Type":"ContainerDied","Data":"30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded"} Nov 03 14:53:15.059130 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.059080 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" event={"ID":"17ba2e92-ab11-4edc-84a0-1339723e2f69","Type":"ContainerDied","Data":"5a417446d9ba45bb7f9306d829b83a75add6e5cc575e14b34042dabcf96289c6"} Nov 03 14:53:15.059130 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.059039 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl" Nov 03 14:53:15.066104 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.066088 4541 scope.go:122] "RemoveContainer" containerID="0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698" Nov 03 14:53:15.066347 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:53:15.066331 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698\": container with ID starting with 0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698 not found: ID does not exist" containerID="0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698" Nov 03 14:53:15.066392 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.066354 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698"} err="failed to get container status \"0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698\": rpc error: code = NotFound desc = could not find container \"0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698\": container with ID starting with 0c974ef11976dd4addbdf9d8cdd9da94705cb924f6b46de75318333c7af41698 not found: ID does not exist" Nov 03 14:53:15.066392 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.066370 4541 scope.go:122] "RemoveContainer" containerID="30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded" Nov 03 14:53:15.075008 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.074991 4541 scope.go:122] "RemoveContainer" containerID="30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded" Nov 03 14:53:15.075271 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:53:15.075255 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded\": container with ID starting with 30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded not found: ID does not exist" containerID="30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded" Nov 03 14:53:15.075333 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.075281 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded"} err="failed to get container status \"30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded\": rpc error: code = NotFound desc = could not find container \"30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded\": container with ID starting with 30b8f4821f2f4d1596002b623b09d92dbb3a90779b45faf61faa1d299eeeaded not found: ID does not exist" Nov 03 14:53:15.213226 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.213200 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl"] Nov 03 14:53:15.217948 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.217925 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfffchrxl"] Nov 03 14:53:15.319909 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.319886 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz"] Nov 03 14:53:15.323982 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.323960 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-21ea69-786cfcjrnz"] Nov 03 14:53:15.444074 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.444040 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz"] Nov 03 14:53:15.444326 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:15.444284 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" containerID="cri-o://b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a" gracePeriod=30 Nov 03 14:53:16.866860 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:16.866829 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ba2e92-ab11-4edc-84a0-1339723e2f69" path="/var/lib/kubelet/pods/17ba2e92-ab11-4edc-84a0-1339723e2f69/volumes" Nov 03 14:53:16.867354 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:16.867329 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dccd27b-49c7-4bf1-8973-f28a0dc13c35" path="/var/lib/kubelet/pods/7dccd27b-49c7-4bf1-8973-f28a0dc13c35/volumes" Nov 03 14:53:16.895995 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:53:16.895966 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:53:20.695521 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.695495 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:53:20.751614 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.751523 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-token\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-token\") pod \"6ac55d86-03ba-416e-9f20-3dc2577d504d\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " Nov 03 14:53:20.751614 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.751596 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-data\") pod \"6ac55d86-03ba-416e-9f20-3dc2577d504d\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " Nov 03 14:53:20.751896 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.751660 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-certs\") pod \"6ac55d86-03ba-416e-9f20-3dc2577d504d\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " Nov 03 14:53:20.751896 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.751701 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-credential-socket\") pod \"6ac55d86-03ba-416e-9f20-3dc2577d504d\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " Nov 03 14:53:20.751896 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.751740 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac55d86-03ba-416e-9f20-3dc2577d504d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac55d86-03ba-416e-9f20-3dc2577d504d-istiod-ca-cert\") pod \"6ac55d86-03ba-416e-9f20-3dc2577d504d\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " Nov 03 14:53:20.751896 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.751766 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-socket\") pod \"6ac55d86-03ba-416e-9f20-3dc2577d504d\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " Nov 03 14:53:20.751896 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.751828 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-envoy\") pod \"6ac55d86-03ba-416e-9f20-3dc2577d504d\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " Nov 03 14:53:20.751896 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.751866 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-podinfo\") pod \"6ac55d86-03ba-416e-9f20-3dc2577d504d\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " Nov 03 14:53:20.752233 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.751933 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-kube-api-access-wcph5\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-kube-api-access-wcph5\") pod \"6ac55d86-03ba-416e-9f20-3dc2577d504d\" (UID: \"6ac55d86-03ba-416e-9f20-3dc2577d504d\") " Nov 03 14:53:20.752233 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.752014 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-certs" pod "6ac55d86-03ba-416e-9f20-3dc2577d504d" (UID: "6ac55d86-03ba-416e-9f20-3dc2577d504d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:53:20.752359 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.752258 4541 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:20.752359 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.752287 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-socket" pod "6ac55d86-03ba-416e-9f20-3dc2577d504d" (UID: "6ac55d86-03ba-416e-9f20-3dc2577d504d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:53:20.752443 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.752391 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac55d86-03ba-416e-9f20-3dc2577d504d-istiod-ca-cert" pod "6ac55d86-03ba-416e-9f20-3dc2577d504d" (UID: "6ac55d86-03ba-416e-9f20-3dc2577d504d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:53:20.752500 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.752433 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-data" pod "6ac55d86-03ba-416e-9f20-3dc2577d504d" (UID: "6ac55d86-03ba-416e-9f20-3dc2577d504d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:53:20.752500 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.752452 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-credential-socket" pod "6ac55d86-03ba-416e-9f20-3dc2577d504d" (UID: "6ac55d86-03ba-416e-9f20-3dc2577d504d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:53:20.754189 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.754162 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-podinfo" pod "6ac55d86-03ba-416e-9f20-3dc2577d504d" (UID: "6ac55d86-03ba-416e-9f20-3dc2577d504d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:53:20.754385 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.754363 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-envoy" pod "6ac55d86-03ba-416e-9f20-3dc2577d504d" (UID: "6ac55d86-03ba-416e-9f20-3dc2577d504d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:53:20.754457 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.754366 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-token" pod "6ac55d86-03ba-416e-9f20-3dc2577d504d" (UID: "6ac55d86-03ba-416e-9f20-3dc2577d504d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:53:20.754594 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.754576 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-kube-api-access-wcph5" pod "6ac55d86-03ba-416e-9f20-3dc2577d504d" (UID: "6ac55d86-03ba-416e-9f20-3dc2577d504d"). InnerVolumeSpecName "kube-api-access-wcph5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:53:20.853097 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.853062 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcph5\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-kube-api-access-wcph5\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:20.853097 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.853095 4541 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-token\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:20.853097 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.853105 4541 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-data\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:20.853343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.853113 4541 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-credential-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:20.853343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.853124 4541 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ac55d86-03ba-416e-9f20-3dc2577d504d-istiod-ca-cert\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:20.853343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.853137 4541 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-workload-socket\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:20.853343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.853147 4541 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-envoy\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:20.853343 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:20.853155 4541 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ac55d86-03ba-416e-9f20-3dc2577d504d-istio-podinfo\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:21.088023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.087938 4541 generic.go:356] "Generic (PLEG): container finished" podID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerID="b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a" exitCode=0 Nov 03 14:53:21.088023 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.087995 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" Nov 03 14:53:21.088256 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.088026 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" event={"ID":"6ac55d86-03ba-416e-9f20-3dc2577d504d","Type":"ContainerDied","Data":"b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a"} Nov 03 14:53:21.088256 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.088087 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz" event={"ID":"6ac55d86-03ba-416e-9f20-3dc2577d504d","Type":"ContainerDied","Data":"a8cc6f03ca7699836a27946b4bd24bc2c4ef3e19181884d2b1c0b33c1f7af1fe"} Nov 03 14:53:21.088256 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.088117 4541 scope.go:122] "RemoveContainer" containerID="b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a" Nov 03 14:53:21.096837 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.096817 4541 scope.go:122] "RemoveContainer" containerID="93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe" Nov 03 14:53:21.104277 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.104253 4541 scope.go:122] "RemoveContainer" containerID="b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a" Nov 03 14:53:21.104550 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:53:21.104530 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a\": container with ID starting with b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a not found: ID does not exist" containerID="b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a" Nov 03 14:53:21.104595 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.104559 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a"} err="failed to get container status \"b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a\": rpc error: code = NotFound desc = could not find container \"b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a\": container with ID starting with b1bbcff6be52a420e452aea67035f5bb7ed6fbf0ef53ef59382669f3c5cc2b3a not found: ID does not exist" Nov 03 14:53:21.104595 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.104578 4541 scope.go:122] "RemoveContainer" containerID="93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe" Nov 03 14:53:21.104794 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:53:21.104766 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe\": container with ID starting with 93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe not found: ID does not exist" containerID="93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe" Nov 03 14:53:21.104838 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.104798 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe"} err="failed to get container status \"93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe\": rpc error: code = NotFound desc = could not find container \"93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe\": container with ID starting with 93406320e3a306db4a593f6a56a0ea1acaef374f5a431da6ee67d57f0b1a50fe not found: ID does not exist" Nov 03 14:53:21.243314 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.243281 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz"] Nov 03 14:53:21.250807 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:21.250765 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-21ea69-openshift-default-684d7746b94fjbz"] Nov 03 14:53:22.866254 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:53:22.866221 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" path="/var/lib/kubelet/pods/6ac55d86-03ba-416e-9f20-3dc2577d504d/volumes" Nov 03 14:53:46.888825 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:53:46.888753 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:54:16.888628 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:54:16.888592 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:54:29.650131 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:54:29.650105 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:54:29.689451 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:54:29.689423 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:54:46.891701 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:54:46.891665 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:55:01.416245 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416212 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv"] Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416589 4541 cpu_manager.go:414] "containerMap: removing container" podUID="7dccd27b-49c7-4bf1-8973-f28a0dc13c35" containerName="payload-pre-processing" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416602 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dccd27b-49c7-4bf1-8973-f28a0dc13c35" containerName="payload-pre-processing" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416620 4541 cpu_manager.go:414] "containerMap: removing container" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416626 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416636 4541 cpu_manager.go:414] "containerMap: removing container" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416641 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416651 4541 cpu_manager.go:414] "containerMap: removing container" podUID="17ba2e92-ab11-4edc-84a0-1339723e2f69" containerName="payload-processing" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416656 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ba2e92-ab11-4edc-84a0-1339723e2f69" containerName="payload-processing" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416710 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416720 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac55d86-03ba-416e-9f20-3dc2577d504d" containerName="istio-proxy" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416726 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ba2e92-ab11-4edc-84a0-1339723e2f69" containerName="payload-processing" Nov 03 14:55:01.416729 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.416735 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dccd27b-49c7-4bf1-8973-f28a0dc13c35" containerName="payload-pre-processing" Nov 03 14:55:01.419651 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.419632 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.471973 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.471942 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 03 14:55:01.519234 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.519194 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wtvwk\"" Nov 03 14:55:01.519429 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.519256 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 03 14:55:01.520804 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.520764 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv"] Nov 03 14:55:01.542353 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.542327 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e57deb9d-44fe-4315-863d-02f14d812148-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-l6jzv\" (UID: \"e57deb9d-44fe-4315-863d-02f14d812148\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.542486 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.542395 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e57deb9d-44fe-4315-863d-02f14d812148-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-l6jzv\" (UID: \"e57deb9d-44fe-4315-863d-02f14d812148\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.542543 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.542473 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md5n7\" (UniqueName: \"kubernetes.io/projected/e57deb9d-44fe-4315-863d-02f14d812148-kube-api-access-md5n7\") pod \"kuadrant-console-plugin-66c4fb575d-l6jzv\" (UID: \"e57deb9d-44fe-4315-863d-02f14d812148\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.643296 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.643263 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e57deb9d-44fe-4315-863d-02f14d812148-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-l6jzv\" (UID: \"e57deb9d-44fe-4315-863d-02f14d812148\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.643473 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.643354 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e57deb9d-44fe-4315-863d-02f14d812148-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-l6jzv\" (UID: \"e57deb9d-44fe-4315-863d-02f14d812148\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.643473 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.643392 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md5n7\" (UniqueName: \"kubernetes.io/projected/e57deb9d-44fe-4315-863d-02f14d812148-kube-api-access-md5n7\") pod \"kuadrant-console-plugin-66c4fb575d-l6jzv\" (UID: \"e57deb9d-44fe-4315-863d-02f14d812148\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.643983 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.643959 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e57deb9d-44fe-4315-863d-02f14d812148-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-l6jzv\" (UID: \"e57deb9d-44fe-4315-863d-02f14d812148\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.645868 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.645847 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e57deb9d-44fe-4315-863d-02f14d812148-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-l6jzv\" (UID: \"e57deb9d-44fe-4315-863d-02f14d812148\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.697974 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.697915 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md5n7\" (UniqueName: \"kubernetes.io/projected/e57deb9d-44fe-4315-863d-02f14d812148-kube-api-access-md5n7\") pod \"kuadrant-console-plugin-66c4fb575d-l6jzv\" (UID: \"e57deb9d-44fe-4315-863d-02f14d812148\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:01.730834 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:01.730808 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" Nov 03 14:55:02.091850 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:02.091825 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv"] Nov 03 14:55:02.094111 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:55:02.094080 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode57deb9d_44fe_4315_863d_02f14d812148.slice/crio-ec97b33c28885239f97d2fa6d12dc2fb6cc6443cb179bce0294f3893f5a37b29 WatchSource:0}: Error finding container ec97b33c28885239f97d2fa6d12dc2fb6cc6443cb179bce0294f3893f5a37b29: Status 404 returned error can't find the container with id ec97b33c28885239f97d2fa6d12dc2fb6cc6443cb179bce0294f3893f5a37b29 Nov 03 14:55:02.547872 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:02.547828 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" event={"ID":"e57deb9d-44fe-4315-863d-02f14d812148","Type":"ContainerStarted","Data":"ec97b33c28885239f97d2fa6d12dc2fb6cc6443cb179bce0294f3893f5a37b29"} Nov 03 14:55:09.612299 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:09.612265 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" event={"ID":"e57deb9d-44fe-4315-863d-02f14d812148","Type":"ContainerStarted","Data":"6e63c8099b9b14ed72c6c73c6c07d10cb01e9770fba57c2733945a6a22510652"} Nov 03 14:55:09.721083 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:09.721032 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-l6jzv" podStartSLOduration=2.19613813 podStartE2EDuration="8.721019464s" totalImagesPullingTime="6.524881334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:55:09.720648384 +0000 UTC m=+6641.034620244" watchObservedRunningTime="2026-11-03 14:55:09.721019464 +0000 UTC m=+6641.034991324" Nov 03 14:55:16.890423 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:55:16.890384 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:55:41.171036 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.171002 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-578f968db4-9mcmb"] Nov 03 14:55:41.190996 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.190967 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.231717 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.231675 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8322e45-5ed0-488b-941f-c3bd8baf169b-openshift-service-ca\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.231916 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.231747 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8322e45-5ed0-488b-941f-c3bd8baf169b-metrics-certs\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.231916 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.231851 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cn5v\" (UniqueName: \"kubernetes.io/projected/d8322e45-5ed0-488b-941f-c3bd8baf169b-kube-api-access-9cn5v\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.231916 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.231893 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8322e45-5ed0-488b-941f-c3bd8baf169b-maas-api-tls\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.289276 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.289241 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-578f968db4-9mcmb"] Nov 03 14:55:41.332581 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.332548 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8322e45-5ed0-488b-941f-c3bd8baf169b-openshift-service-ca\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.332728 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.332599 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8322e45-5ed0-488b-941f-c3bd8baf169b-metrics-certs\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.332728 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.332629 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cn5v\" (UniqueName: \"kubernetes.io/projected/d8322e45-5ed0-488b-941f-c3bd8baf169b-kube-api-access-9cn5v\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.332728 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.332651 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8322e45-5ed0-488b-941f-c3bd8baf169b-maas-api-tls\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.333358 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.333331 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8322e45-5ed0-488b-941f-c3bd8baf169b-openshift-service-ca\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.335269 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.335247 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8322e45-5ed0-488b-941f-c3bd8baf169b-maas-api-tls\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.335349 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.335247 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8322e45-5ed0-488b-941f-c3bd8baf169b-metrics-certs\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.387284 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.387260 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cn5v\" (UniqueName: \"kubernetes.io/projected/d8322e45-5ed0-488b-941f-c3bd8baf169b-kube-api-access-9cn5v\") pod \"maas-api-578f968db4-9mcmb\" (UID: \"d8322e45-5ed0-488b-941f-c3bd8baf169b\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.504540 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.504475 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:41.865701 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.865679 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-578f968db4-9mcmb"] Nov 03 14:55:41.867520 ip-10-0-132-174 kubenswrapper[4541]: W1103 14:55:41.867494 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8322e45_5ed0_488b_941f_c3bd8baf169b.slice/crio-d0da8c8dec77fbe929b7db14e9691c4eacdb5417063f2acafd88e5eba296ebb2 WatchSource:0}: Error finding container d0da8c8dec77fbe929b7db14e9691c4eacdb5417063f2acafd88e5eba296ebb2: Status 404 returned error can't find the container with id d0da8c8dec77fbe929b7db14e9691c4eacdb5417063f2acafd88e5eba296ebb2 Nov 03 14:55:41.868931 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:41.868914 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:55:42.783197 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:42.783152 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" event={"ID":"d8322e45-5ed0-488b-941f-c3bd8baf169b","Type":"ContainerStarted","Data":"d0da8c8dec77fbe929b7db14e9691c4eacdb5417063f2acafd88e5eba296ebb2"} Nov 03 14:55:43.788531 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:43.788497 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" event={"ID":"d8322e45-5ed0-488b-941f-c3bd8baf169b","Type":"ContainerStarted","Data":"f0ae924a6776ee709ea1b86db46141752f6c923b021eeaee770e05f7532ac814"} Nov 03 14:55:43.788959 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:43.788609 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:43.899065 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:43.899013 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" podStartSLOduration=1.84614637 podStartE2EDuration="2.899000192s" totalImagesPullingTime="1.052853822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:55:43.897279293 +0000 UTC m=+6675.211251164" watchObservedRunningTime="2026-11-03 14:55:43.899000192 +0000 UTC m=+6675.212972076" Nov 03 14:55:46.888496 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:55:46.888460 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:55:49.798333 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:49.798303 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-578f968db4-9mcmb" Nov 03 14:55:49.956204 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:49.956162 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm"] Nov 03 14:55:49.958271 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:49.956369 4541 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" podUID="0094c30d-13f8-4f1d-b51a-69260417fe0b" containerName="maas-api" containerID="cri-o://7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f" gracePeriod=30 Nov 03 14:55:51.203535 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.203508 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 14:55:51.318757 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.318678 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0094c30d-13f8-4f1d-b51a-69260417fe0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094c30d-13f8-4f1d-b51a-69260417fe0b-openshift-service-ca\") pod \"0094c30d-13f8-4f1d-b51a-69260417fe0b\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " Nov 03 14:55:51.318757 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.318734 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0094c30d-13f8-4f1d-b51a-69260417fe0b-kube-api-access-4wg8f\" (UniqueName: \"kubernetes.io/projected/0094c30d-13f8-4f1d-b51a-69260417fe0b-kube-api-access-4wg8f\") pod \"0094c30d-13f8-4f1d-b51a-69260417fe0b\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " Nov 03 14:55:51.319057 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.318763 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-maas-api-tls\") pod \"0094c30d-13f8-4f1d-b51a-69260417fe0b\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " Nov 03 14:55:51.319057 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.318816 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-metrics-certs\") pod \"0094c30d-13f8-4f1d-b51a-69260417fe0b\" (UID: \"0094c30d-13f8-4f1d-b51a-69260417fe0b\") " Nov 03 14:55:51.319195 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.319102 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0094c30d-13f8-4f1d-b51a-69260417fe0b-openshift-service-ca" pod "0094c30d-13f8-4f1d-b51a-69260417fe0b" (UID: "0094c30d-13f8-4f1d-b51a-69260417fe0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:55:51.319542 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.319504 4541 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0094c30d-13f8-4f1d-b51a-69260417fe0b-openshift-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:51.321033 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.321004 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0094c30d-13f8-4f1d-b51a-69260417fe0b-kube-api-access-4wg8f" pod "0094c30d-13f8-4f1d-b51a-69260417fe0b" (UID: "0094c30d-13f8-4f1d-b51a-69260417fe0b"). InnerVolumeSpecName "kube-api-access-4wg8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:55:51.321210 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.321170 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-maas-api-tls" pod "0094c30d-13f8-4f1d-b51a-69260417fe0b" (UID: "0094c30d-13f8-4f1d-b51a-69260417fe0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:55:51.321210 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.321189 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-metrics-certs" pod "0094c30d-13f8-4f1d-b51a-69260417fe0b" (UID: "0094c30d-13f8-4f1d-b51a-69260417fe0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:55:51.420751 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.420724 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wg8f\" (UniqueName: \"kubernetes.io/projected/0094c30d-13f8-4f1d-b51a-69260417fe0b-kube-api-access-4wg8f\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:51.420751 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.420750 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-maas-api-tls\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:51.420908 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.420761 4541 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0094c30d-13f8-4f1d-b51a-69260417fe0b-metrics-certs\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:51.823894 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.823853 4541 generic.go:356] "Generic (PLEG): container finished" podID="0094c30d-13f8-4f1d-b51a-69260417fe0b" containerID="7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f" exitCode=0 Nov 03 14:55:51.824077 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.823905 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" Nov 03 14:55:51.824077 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.823934 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" event={"ID":"0094c30d-13f8-4f1d-b51a-69260417fe0b","Type":"ContainerDied","Data":"7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f"} Nov 03 14:55:51.824077 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.823992 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm" event={"ID":"0094c30d-13f8-4f1d-b51a-69260417fe0b","Type":"ContainerDied","Data":"d7a45f70b270f7743b84eff82b8dde26ba568b0d97014701abab4b6338d197e2"} Nov 03 14:55:51.824077 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.824014 4541 scope.go:122] "RemoveContainer" containerID="7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f" Nov 03 14:55:51.833060 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.833041 4541 scope.go:122] "RemoveContainer" containerID="7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f" Nov 03 14:55:51.833345 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:55:51.833321 4541 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f\": container with ID starting with 7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f not found: ID does not exist" containerID="7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f" Nov 03 14:55:51.833419 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.833353 4541 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f"} err="failed to get container status \"7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f\": rpc error: code = NotFound desc = could not find container \"7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f\": container with ID starting with 7c15a30c713442ae0927f5a27aef93d016690665b19435bbf9fb2bf7360c157f not found: ID does not exist" Nov 03 14:55:51.977949 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:51.977913 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm"] Nov 03 14:55:52.024902 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:52.024867 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-bc7hm"] Nov 03 14:55:52.866073 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:55:52.866041 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0094c30d-13f8-4f1d-b51a-69260417fe0b" path="/var/lib/kubelet/pods/0094c30d-13f8-4f1d-b51a-69260417fe0b/volumes" Nov 03 14:56:16.888476 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:56:16.888394 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:56:46.888975 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:56:46.888938 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:57:14.147956 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:57:14.147912 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-68h6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:57:14.148757 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:57:14.148718 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-68h6d" podUID="06cc0694-c336-47b3-b0c8-856a3c2441c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:57:16.888790 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:57:16.888739 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:57:46.888581 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:57:46.888543 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:58:16.888756 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:58:16.888720 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:58:28.450991 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:58:28.450948 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:58:28.451678 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:58:28.451641 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:58:46.888370 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:58:46.888326 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:59:16.889138 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:59:16.889057 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 14:59:29.673761 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:59:29.673730 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:59:29.715213 ip-10-0-132-174 kubenswrapper[4541]: I1103 14:59:29.715185 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5sjz9_f9f28189-a808-48a4-b2da-d1950b84c7d0/ovn-acl-logging/0.log" Nov 03 14:59:46.888604 ip-10-0-132-174 kubenswrapper[4541]: E1103 14:59:46.888568 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 15:00:00.161558 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.161523 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6"] Nov 03 15:00:00.162211 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.162094 4541 cpu_manager.go:414] "containerMap: removing container" podUID="0094c30d-13f8-4f1d-b51a-69260417fe0b" containerName="maas-api" Nov 03 15:00:00.162211 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.162127 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0094c30d-13f8-4f1d-b51a-69260417fe0b" containerName="maas-api" Nov 03 15:00:00.162340 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.162233 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="0094c30d-13f8-4f1d-b51a-69260417fe0b" containerName="maas-api" Nov 03 15:00:00.165148 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.165126 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:00.221094 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.221061 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6"] Nov 03 15:00:00.262371 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.262343 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sshv8\" (UniqueName: \"kubernetes.io/projected/bb126f02-1cda-4819-a0a3-b7aa7338b951-kube-api-access-sshv8\") pod \"maas-api-key-cleanup-29895300-nhrd6\" (UID: \"bb126f02-1cda-4819-a0a3-b7aa7338b951\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:00.262520 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.262384 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb126f02-1cda-4819-a0a3-b7aa7338b951-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895300-nhrd6\" (UID: \"bb126f02-1cda-4819-a0a3-b7aa7338b951\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:00.263298 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.263277 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 15:00:00.363224 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.363197 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sshv8\" (UniqueName: \"kubernetes.io/projected/bb126f02-1cda-4819-a0a3-b7aa7338b951-kube-api-access-sshv8\") pod \"maas-api-key-cleanup-29895300-nhrd6\" (UID: \"bb126f02-1cda-4819-a0a3-b7aa7338b951\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:00.363377 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.363239 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb126f02-1cda-4819-a0a3-b7aa7338b951-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895300-nhrd6\" (UID: \"bb126f02-1cda-4819-a0a3-b7aa7338b951\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:00.363843 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.363825 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb126f02-1cda-4819-a0a3-b7aa7338b951-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895300-nhrd6\" (UID: \"bb126f02-1cda-4819-a0a3-b7aa7338b951\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:00.417331 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.417265 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sshv8\" (UniqueName: \"kubernetes.io/projected/bb126f02-1cda-4819-a0a3-b7aa7338b951-kube-api-access-sshv8\") pod \"maas-api-key-cleanup-29895300-nhrd6\" (UID: \"bb126f02-1cda-4819-a0a3-b7aa7338b951\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:00.478851 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.478825 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:00.850482 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:00.842453 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6"] Nov 03 15:00:01.115661 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:01.115578 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" event={"ID":"bb126f02-1cda-4819-a0a3-b7aa7338b951","Type":"ContainerStarted","Data":"6c03e5eec64c0d4b19950c03c561208d6e369b89cc34b7448475e511ca6743e8"} Nov 03 15:00:02.121014 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:02.120979 4541 generic.go:356] "Generic (PLEG): container finished" podID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerID="c9834b52c09886e2ce96808261d7873bd50e019c6cb64e36c022e1258c23efda" exitCode=60 Nov 03 15:00:02.121393 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:02.121033 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" event={"ID":"bb126f02-1cda-4819-a0a3-b7aa7338b951","Type":"ContainerDied","Data":"c9834b52c09886e2ce96808261d7873bd50e019c6cb64e36c022e1258c23efda"} Nov 03 15:00:02.121393 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:02.121324 4541 scope.go:122] "RemoveContainer" containerID="c9834b52c09886e2ce96808261d7873bd50e019c6cb64e36c022e1258c23efda" Nov 03 15:00:03.127393 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:03.127353 4541 generic.go:356] "Generic (PLEG): container finished" podID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerID="1f40c8680ca701c1f8b2a1a5e14a5de278ae6c572db55ecc4692e4b8cfcb9fdf" exitCode=60 Nov 03 15:00:03.127820 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:03.127436 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" event={"ID":"bb126f02-1cda-4819-a0a3-b7aa7338b951","Type":"ContainerDied","Data":"1f40c8680ca701c1f8b2a1a5e14a5de278ae6c572db55ecc4692e4b8cfcb9fdf"} Nov 03 15:00:03.127820 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:03.127502 4541 scope.go:122] "RemoveContainer" containerID="c9834b52c09886e2ce96808261d7873bd50e019c6cb64e36c022e1258c23efda" Nov 03 15:00:03.127967 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:03.127824 4541 scope.go:122] "RemoveContainer" containerID="1f40c8680ca701c1f8b2a1a5e14a5de278ae6c572db55ecc4692e4b8cfcb9fdf" Nov 03 15:00:03.128073 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:00:03.128050 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895300-nhrd6_odh-ai-gateway-infra(bb126f02-1cda-4819-a0a3-b7aa7338b951)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" Nov 03 15:00:04.135129 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:04.135093 4541 scope.go:122] "RemoveContainer" containerID="1f40c8680ca701c1f8b2a1a5e14a5de278ae6c572db55ecc4692e4b8cfcb9fdf" Nov 03 15:00:04.135528 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:00:04.135376 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895300-nhrd6_odh-ai-gateway-infra(bb126f02-1cda-4819-a0a3-b7aa7338b951)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" Nov 03 15:00:14.874795 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:14.874512 4541 scope.go:122] "RemoveContainer" containerID="1f40c8680ca701c1f8b2a1a5e14a5de278ae6c572db55ecc4692e4b8cfcb9fdf" Nov 03 15:00:16.223699 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:16.223620 4541 generic.go:356] "Generic (PLEG): container finished" podID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerID="de4d16a81fc202cd906f76090a98616d8b7bd53aac64c6cb392506b291f6e85a" exitCode=60 Nov 03 15:00:16.224167 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:16.223692 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" event={"ID":"bb126f02-1cda-4819-a0a3-b7aa7338b951","Type":"ContainerDied","Data":"de4d16a81fc202cd906f76090a98616d8b7bd53aac64c6cb392506b291f6e85a"} Nov 03 15:00:16.224167 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:16.223752 4541 scope.go:122] "RemoveContainer" containerID="1f40c8680ca701c1f8b2a1a5e14a5de278ae6c572db55ecc4692e4b8cfcb9fdf" Nov 03 15:00:16.224167 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:16.224075 4541 scope.go:122] "RemoveContainer" containerID="de4d16a81fc202cd906f76090a98616d8b7bd53aac64c6cb392506b291f6e85a" Nov 03 15:00:16.224313 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:00:16.224291 4541 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895300-nhrd6_odh-ai-gateway-infra(bb126f02-1cda-4819-a0a3-b7aa7338b951)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" Nov 03 15:00:16.888028 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:00:16.887982 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 15:00:17.345971 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:17.345939 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6"] Nov 03 15:00:17.472409 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:17.472385 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:17.536413 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:17.536388 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb126f02-1cda-4819-a0a3-b7aa7338b951-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb126f02-1cda-4819-a0a3-b7aa7338b951-maas-api-service-ca\") pod \"bb126f02-1cda-4819-a0a3-b7aa7338b951\" (UID: \"bb126f02-1cda-4819-a0a3-b7aa7338b951\") " Nov 03 15:00:17.536557 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:17.536425 4541 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb126f02-1cda-4819-a0a3-b7aa7338b951-kube-api-access-sshv8\" (UniqueName: \"kubernetes.io/projected/bb126f02-1cda-4819-a0a3-b7aa7338b951-kube-api-access-sshv8\") pod \"bb126f02-1cda-4819-a0a3-b7aa7338b951\" (UID: \"bb126f02-1cda-4819-a0a3-b7aa7338b951\") " Nov 03 15:00:17.536705 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:17.536682 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb126f02-1cda-4819-a0a3-b7aa7338b951-maas-api-service-ca" pod "bb126f02-1cda-4819-a0a3-b7aa7338b951" (UID: "bb126f02-1cda-4819-a0a3-b7aa7338b951"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 15:00:17.538338 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:17.538312 4541 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb126f02-1cda-4819-a0a3-b7aa7338b951-kube-api-access-sshv8" pod "bb126f02-1cda-4819-a0a3-b7aa7338b951" (UID: "bb126f02-1cda-4819-a0a3-b7aa7338b951"). InnerVolumeSpecName "kube-api-access-sshv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 15:00:17.637227 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:17.637151 4541 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb126f02-1cda-4819-a0a3-b7aa7338b951-maas-api-service-ca\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:17.637227 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:17.637178 4541 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sshv8\" (UniqueName: \"kubernetes.io/projected/bb126f02-1cda-4819-a0a3-b7aa7338b951-kube-api-access-sshv8\") on node \"ip-10-0-132-174.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 15:00:18.236878 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:18.236846 4541 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" Nov 03 15:00:18.236878 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:18.236861 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6" event={"ID":"bb126f02-1cda-4819-a0a3-b7aa7338b951","Type":"ContainerDied","Data":"6c03e5eec64c0d4b19950c03c561208d6e369b89cc34b7448475e511ca6743e8"} Nov 03 15:00:18.237099 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:18.236930 4541 scope.go:122] "RemoveContainer" containerID="de4d16a81fc202cd906f76090a98616d8b7bd53aac64c6cb392506b291f6e85a" Nov 03 15:00:18.393106 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:18.393068 4541 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6"] Nov 03 15:00:18.439747 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:18.439713 4541 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895300-nhrd6"] Nov 03 15:00:18.866345 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:18.866315 4541 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" path="/var/lib/kubelet/pods/bb126f02-1cda-4819-a0a3-b7aa7338b951/volumes" Nov 03 15:00:28.455972 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:28.455910 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:00:28.456663 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:28.456613 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jkh4x" podUID="7828782e-ff3e-4876-b530-af687a56b074" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:00:46.897433 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:00:46.897385 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 15:00:59.964968 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:59.964922 4541 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:00:59.967337 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:00:59.964997 4541 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-m7hd7" podUID="b94a6140-9ce0-476f-9385-a2756e76c1b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:01:16.890642 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:01:16.890603 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 15:01:46.888546 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:01:46.888507 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 15:02:07.952661 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:07.952622 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f5d89c4f6-jtgnc_cef8280b-0119-4e61-94c5-5ddea15d4f04/authorino/0.log" Nov 03 15:02:16.889537 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:02:16.889451 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 15:02:19.841165 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:19.841129 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cb4ffdf47-czmlj_bb24765f-1472-41e4-863d-ecdcc9f1ee97/manager/0.log" Nov 03 15:02:26.680147 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:26.680114 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-578f968db4-9mcmb_d8322e45-5ed0-488b-941f-c3bd8baf169b/maas-api/0.log" Nov 03 15:02:32.570965 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:32.570926 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f5d89c4f6-jtgnc_cef8280b-0119-4e61-94c5-5ddea15d4f04/authorino/0.log" Nov 03 15:02:33.437699 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:33.437669 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-l6jzv_e57deb9d-44fe-4315-863d-02f14d812148/kuadrant-console-plugin/0.log" Nov 03 15:02:36.134497 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:36.134459 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77875f86b8-n5psr_f8c2eb10-7803-4a98-9859-1ab515a46bce/kube-auth-proxy/0.log" Nov 03 15:02:36.354302 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:36.354268 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-68h6d_06cc0694-c336-47b3-b0c8-856a3c2441c0/istio-proxy/0.log" Nov 03 15:02:37.732366 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:37.732325 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-jkh4x_7828782e-ff3e-4876-b530-af687a56b074/istio-proxy/0.log" Nov 03 15:02:38.321314 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:38.321283 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-m7hd7_b94a6140-9ce0-476f-9385-a2756e76c1b2/istio-proxy/0.log" Nov 03 15:02:39.494947 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:39.494915 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7c88cb685f-zjswf_234c192e-ace3-483d-9a9b-6d92aae5d15b/payload-processing/0.log" Nov 03 15:02:40.772040 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:40.772007 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-z7mb8_ccd2faed-c412-486e-ba4e-8d103fc677e4/main/0.log" Nov 03 15:02:41.038495 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:41.038414 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-zcx79_9f4c2084-9be1-422b-a365-fe3c08cd968c/main/0.log" Nov 03 15:02:41.307259 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:41.307182 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-rthrd_9768d8a0-31ff-432d-bb2f-875f4077ea2a/main/0.log" Nov 03 15:02:41.527049 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:41.527023 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-tv7mv_987205e0-22af-461b-9c75-ec19537ebaa3/main/0.log" Nov 03 15:02:42.031419 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:42.031384 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-6xft2_0474dd31-37ce-4eb3-9c8c-336d9a19953b/main/0.log" Nov 03 15:02:42.547142 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:42.547115 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-95zt5_e961fb1a-50db-4f9b-9a17-b690246f59a7/main/0.log" Nov 03 15:02:46.888727 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:02:46.888687 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 15:02:53.761660 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.761620 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-75lvz/must-gather-7l6bb"] Nov 03 15:02:53.762131 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.762090 4541 cpu_manager.go:414] "containerMap: removing container" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerName="cleanup" Nov 03 15:02:53.762131 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.762109 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerName="cleanup" Nov 03 15:02:53.762205 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.762134 4541 cpu_manager.go:414] "containerMap: removing container" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerName="cleanup" Nov 03 15:02:53.762205 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.762140 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerName="cleanup" Nov 03 15:02:53.762205 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.762198 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerName="cleanup" Nov 03 15:02:53.762301 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.762208 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerName="cleanup" Nov 03 15:02:53.762301 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.762262 4541 cpu_manager.go:414] "containerMap: removing container" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerName="cleanup" Nov 03 15:02:53.762301 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.762268 4541 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerName="cleanup" Nov 03 15:02:53.762392 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.762330 4541 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb126f02-1cda-4819-a0a3-b7aa7338b951" containerName="cleanup" Nov 03 15:02:53.765474 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.765454 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-75lvz/must-gather-7l6bb" Nov 03 15:02:53.863734 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.863700 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-75lvz\"/\"openshift-service-ca.crt\"" Nov 03 15:02:53.863927 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.863809 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-75lvz\"/\"kube-root-ca.crt\"" Nov 03 15:02:53.863927 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.863882 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-75lvz\"/\"default-dockercfg-cnsx8\"" Nov 03 15:02:53.865520 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.865501 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-75lvz/must-gather-7l6bb"] Nov 03 15:02:53.897001 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.896964 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5kwd\" (UniqueName: \"kubernetes.io/projected/c6e7a6e2-2baf-4bc9-9980-770bade10799-kube-api-access-j5kwd\") pod \"must-gather-7l6bb\" (UID: \"c6e7a6e2-2baf-4bc9-9980-770bade10799\") " pod="openshift-must-gather-75lvz/must-gather-7l6bb" Nov 03 15:02:53.897172 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.897016 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c6e7a6e2-2baf-4bc9-9980-770bade10799-must-gather-output\") pod \"must-gather-7l6bb\" (UID: \"c6e7a6e2-2baf-4bc9-9980-770bade10799\") " pod="openshift-must-gather-75lvz/must-gather-7l6bb" Nov 03 15:02:53.998043 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.998003 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5kwd\" (UniqueName: \"kubernetes.io/projected/c6e7a6e2-2baf-4bc9-9980-770bade10799-kube-api-access-j5kwd\") pod \"must-gather-7l6bb\" (UID: \"c6e7a6e2-2baf-4bc9-9980-770bade10799\") " pod="openshift-must-gather-75lvz/must-gather-7l6bb" Nov 03 15:02:53.998249 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.998059 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c6e7a6e2-2baf-4bc9-9980-770bade10799-must-gather-output\") pod \"must-gather-7l6bb\" (UID: \"c6e7a6e2-2baf-4bc9-9980-770bade10799\") " pod="openshift-must-gather-75lvz/must-gather-7l6bb" Nov 03 15:02:53.998534 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:53.998509 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c6e7a6e2-2baf-4bc9-9980-770bade10799-must-gather-output\") pod \"must-gather-7l6bb\" (UID: \"c6e7a6e2-2baf-4bc9-9980-770bade10799\") " pod="openshift-must-gather-75lvz/must-gather-7l6bb" Nov 03 15:02:54.052746 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:54.052671 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5kwd\" (UniqueName: \"kubernetes.io/projected/c6e7a6e2-2baf-4bc9-9980-770bade10799-kube-api-access-j5kwd\") pod \"must-gather-7l6bb\" (UID: \"c6e7a6e2-2baf-4bc9-9980-770bade10799\") " pod="openshift-must-gather-75lvz/must-gather-7l6bb" Nov 03 15:02:54.079350 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:54.079300 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-75lvz/must-gather-7l6bb" Nov 03 15:02:54.445059 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:54.445029 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-75lvz/must-gather-7l6bb"] Nov 03 15:02:54.447164 ip-10-0-132-174 kubenswrapper[4541]: W1103 15:02:54.447133 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6e7a6e2_2baf_4bc9_9980_770bade10799.slice/crio-772c9dee3c8fc9478631d71b3b7a6c436ab635cce6f46b0332c4c1f260fea150 WatchSource:0}: Error finding container 772c9dee3c8fc9478631d71b3b7a6c436ab635cce6f46b0332c4c1f260fea150: Status 404 returned error can't find the container with id 772c9dee3c8fc9478631d71b3b7a6c436ab635cce6f46b0332c4c1f260fea150 Nov 03 15:02:54.449045 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:54.449027 4541 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 15:02:55.143964 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:02:55.143929 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-75lvz/must-gather-7l6bb" event={"ID":"c6e7a6e2-2baf-4bc9-9980-770bade10799","Type":"ContainerStarted","Data":"772c9dee3c8fc9478631d71b3b7a6c436ab635cce6f46b0332c4c1f260fea150"} Nov 03 15:03:03.792075 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:03.792012 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ncrq2_89e41adf-23e2-4eaa-a335-59002579bdce/global-pull-secret-syncer/0.log" Nov 03 15:03:04.199213 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:04.199173 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-75lvz/must-gather-7l6bb" event={"ID":"c6e7a6e2-2baf-4bc9-9980-770bade10799","Type":"ContainerStarted","Data":"7df04447c4c3fb50543e7538a771463d065e0f9dfddfa1bffc79fe5531435ee0"} Nov 03 15:03:04.199392 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:04.199225 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-75lvz/must-gather-7l6bb" event={"ID":"c6e7a6e2-2baf-4bc9-9980-770bade10799","Type":"ContainerStarted","Data":"3456c37234b2e108d9a5996671e6ebf3a9c21910aa12591970c3e76884d9b505"} Nov 03 15:03:04.309372 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:04.309320 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-75lvz/must-gather-7l6bb" podStartSLOduration=2.30584258 podStartE2EDuration="11.309306461s" totalImagesPullingTime="9.003463881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 15:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:03:04.308121323 +0000 UTC m=+7115.622093184" watchObservedRunningTime="2026-11-03 15:03:04.309306461 +0000 UTC m=+7115.623278321" Nov 03 15:03:06.672257 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:06.672220 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pjnlk_91090c0c-bbfd-4e9a-b7f0-4bf7e50027e8/konnectivity-agent/0.log" Nov 03 15:03:07.671188 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:07.671151 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cb4ffdf47-czmlj_bb24765f-1472-41e4-863d-ecdcc9f1ee97/manager/0.log" Nov 03 15:03:07.825835 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:07.825800 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-174.us-west-2.compute.internal_3ac4092c55873e071f156a04a17c98a5/haproxy/0.log" Nov 03 15:03:11.708761 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:11.708727 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f5d89c4f6-jtgnc_cef8280b-0119-4e61-94c5-5ddea15d4f04/authorino/0.log" Nov 03 15:03:13.092603 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.092565 4541 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w"] Nov 03 15:03:13.099397 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.099373 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.148245 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.148212 4541 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wn4x5\"/\"default-dockercfg-gmmrj\"" Nov 03 15:03:13.148483 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.148442 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wn4x5\"/\"kube-root-ca.crt\"" Nov 03 15:03:13.181982 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.181943 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl65f\" (UniqueName: \"kubernetes.io/projected/450132e5-e0eb-4f84-8b44-f8294b7ac242-kube-api-access-fl65f\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.181982 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.181990 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-podres\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.182238 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.182088 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-sys\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.182238 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.182146 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-lib-modules\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.182238 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.182165 4541 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-proc\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.195304 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.195275 4541 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wn4x5\"/\"openshift-service-ca.crt\"" Nov 03 15:03:13.197175 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.197149 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w"] Nov 03 15:03:13.283422 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.283384 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl65f\" (UniqueName: \"kubernetes.io/projected/450132e5-e0eb-4f84-8b44-f8294b7ac242-kube-api-access-fl65f\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.283422 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.283424 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-podres\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.283677 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.283463 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-sys\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.283677 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.283537 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-lib-modules\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.283677 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.283566 4541 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-proc\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.283677 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.283578 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-podres\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.283677 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.283651 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-proc\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.283677 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.283666 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-lib-modules\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.283677 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.283651 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/450132e5-e0eb-4f84-8b44-f8294b7ac242-sys\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.337488 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.337448 4541 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl65f\" (UniqueName: \"kubernetes.io/projected/450132e5-e0eb-4f84-8b44-f8294b7ac242-kube-api-access-fl65f\") pod \"perf-node-gather-daemonset-65b6w\" (UID: \"450132e5-e0eb-4f84-8b44-f8294b7ac242\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.412850 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.412817 4541 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:13.728272 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.728194 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-l6jzv_e57deb9d-44fe-4315-863d-02f14d812148/kuadrant-console-plugin/0.log" Nov 03 15:03:13.783821 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:13.783790 4541 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w"] Nov 03 15:03:13.785242 ip-10-0-132-174 kubenswrapper[4541]: W1103 15:03:13.785215 4541 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod450132e5_e0eb_4f84_8b44_f8294b7ac242.slice/crio-cd7ac60785d03470da0e0ff589fb4e925d0f6ebec679a2563eb31e000c05057d WatchSource:0}: Error finding container cd7ac60785d03470da0e0ff589fb4e925d0f6ebec679a2563eb31e000c05057d: Status 404 returned error can't find the container with id cd7ac60785d03470da0e0ff589fb4e925d0f6ebec679a2563eb31e000c05057d Nov 03 15:03:14.250551 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:14.250516 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" event={"ID":"450132e5-e0eb-4f84-8b44-f8294b7ac242","Type":"ContainerStarted","Data":"6d20a8e1f83adbd3244f2c173d2a4425f5e3203adcdaa163afdd84acc1cb3b36"} Nov 03 15:03:14.251012 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:14.250572 4541 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" event={"ID":"450132e5-e0eb-4f84-8b44-f8294b7ac242","Type":"ContainerStarted","Data":"cd7ac60785d03470da0e0ff589fb4e925d0f6ebec679a2563eb31e000c05057d"} Nov 03 15:03:14.251012 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:14.250595 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:14.363353 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:14.359748 4541 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" podStartSLOduration=1.359731413 podStartE2EDuration="1.359731413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:03:14.357513627 +0000 UTC m=+7125.671485487" watchObservedRunningTime="2026-11-03 15:03:14.359731413 +0000 UTC m=+7125.673703273" Nov 03 15:03:14.486059 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:14.486022 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cb4ffdf47-czmlj_bb24765f-1472-41e4-863d-ecdcc9f1ee97/manager/0.log" Nov 03 15:03:16.899638 ip-10-0-132-174 kubenswrapper[4541]: E1103 15:03:16.899596 4541 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-pwsr8" containerName="extractor" Nov 03 15:03:20.266662 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:20.266634 4541 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-65b6w" Nov 03 15:03:20.551264 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:20.551180 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-cb4ffdf47-czmlj_bb24765f-1472-41e4-863d-ecdcc9f1ee97/manager/0.log" Nov 03 15:03:23.110038 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:23.109999 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sdmbx_01cfefb2-c2f1-4677-a477-e9a7e9afb27d/node-exporter/0.log" Nov 03 15:03:23.505614 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:23.505554 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sdmbx_01cfefb2-c2f1-4677-a477-e9a7e9afb27d/kube-rbac-proxy/0.log" Nov 03 15:03:23.897911 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:23.897836 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sdmbx_01cfefb2-c2f1-4677-a477-e9a7e9afb27d/init-textfile/0.log" Nov 03 15:03:28.906014 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:28.905976 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77875f86b8-n5psr_f8c2eb10-7803-4a98-9859-1ab515a46bce/kube-auth-proxy/0.log" Nov 03 15:03:29.124037 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:29.124005 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-68h6d_06cc0694-c336-47b3-b0c8-856a3c2441c0/istio-proxy/0.log" Nov 03 15:03:30.211216 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:30.211180 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-jkh4x_7828782e-ff3e-4876-b530-af687a56b074/istio-proxy/0.log" Nov 03 15:03:30.642729 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:30.642647 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-m7hd7_b94a6140-9ce0-476f-9385-a2756e76c1b2/istio-proxy/0.log" Nov 03 15:03:30.943661 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:30.943631 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2kn64_8c02bcc5-31fc-4914-b0f8-605b0700aa8c/prometheus-operator/0.log" Nov 03 15:03:31.327199 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:31.327117 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-2kn64_8c02bcc5-31fc-4914-b0f8-605b0700aa8c/kube-rbac-proxy/0.log" Nov 03 15:03:31.718647 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:31.718615 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7c88cb685f-zjswf_234c192e-ace3-483d-9a9b-6d92aae5d15b/payload-processing/0.log" Nov 03 15:03:31.722515 ip-10-0-132-174 kubenswrapper[4541]: I1103 15:03:31.722488 4541 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-2fp88_bea2f07b-470d-4c61-8740-989b36f04c36/prometheus-operator-admission-webhook/0.log"